9091a1ae-29be-458c-984a-339d213919c4

Committed 12 Dec 2024 07:41PM UTC coverage: 26.434% (-69.7%) from 96.113%

Build # 9091a1ae-29be-458c-984a-339d213919c4

Build Type

Pull #2000

circleci

Committed by

jrgriffiniii

Commit Message

Removing integration with ActiveStorage

Pull Request Pull Request #2000: Bump actionpack from 7.2.1.1 to 7.2.2.1

Run Details

945 of 3575 relevant lines covered (26.43%)

0.35 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

38.89

/app/services/file_rename_service.rb

# frozen_string_literal: true

# We sometimes have data with filenames that contain characters that AWS S3 cannot handle. In those cases we want to:
# 1.  Rename the files to something that is AWS legal. Replace all illegal characters with a _ (underscore)
# 2.  Ensure there are no duplicate file names after the renaming by appending a (1), (2) at the end of the filename
#     if the file has been renamed
# 3.  Keep a record of all of the file names as they originally existed and what they were renamed to
# 4.  The record goes into a file called files_renamed.txt, which contains a list of all files that have been renamed
#     and what they were renamed to, along with a timestamp
# 5.  This files_renamed.txt file gets added to the dataset as a payload file, akin to a README.txt or license.txt
class FileRenameService
  # See this reference for the full list of characters that cannot be used in filenames for AWS S3:
  # https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html
  # This service will only attempt to fix the most likely problems. For example, we will not try to
  # handle "ASCII character ranges 00–1F hex (0–31 decimal) and 7F (127 decimal)"
  # Note that we do not rename for a single space, but we do for two spaces together, because according to S3 docs:
  # "Significant sequences of spaces might be lost in some uses (especially multiple spaces)"
  ILLEGAL_CHARACTERS = [
    "&", "$", "@", "=", ";", ":", "+", "  ", ",", "?", "\\", "{", "}", "^", "%", "`", "[", "]", "'", '"', ">", "<", "~", "#", "|"
  ].freeze

  attr_reader :original_filename

  def initialize(filename:)
    @original_filename = filename
  end

  def needs_rename?
    @needs_rename ||= check_if_file_needs_rename
  end

  def check_if_file_needs_rename
    ILLEGAL_CHARACTERS.each do |char|
      return true if @original_filename.include? char
    end
    false
  end

  # Replace every instance of an illegal character with an underscore.
  # Append an index number in parentheses just before the file extension,
  # so we avoid ever accidentally naming two files identically and causing
  # one to over-write the other.
  def new_filename(index)
    nf = @original_filename.dup
    ILLEGAL_CHARACTERS.each do |char|
      nf.gsub!(char, "_")
    end
    split = nf.split(".")
    split[-2] = "#{split[-2]}(#{index})"
    split.join(".")
  end
end

1	# frozen_string_literal: true
2
3	# We sometimes have data with filenames that contain characters that AWS S3 cannot handle. In those cases we want to:
4	# 1. Rename the files to something that is AWS legal. Replace all illegal characters with a _ (underscore)
5	# 2. Ensure there are no duplicate file names after the renaming by appending a (1), (2) at the end of the filename
6	# if the file has been renamed
7	# 3. Keep a record of all of the file names as they originally existed and what they were renamed to
8	# 4. The record goes into a file called files_renamed.txt, which contains a list of all files that have been renamed
9	# and what they were renamed to, along with a timestamp
10	# 5. This files_renamed.txt file gets added to the dataset as a payload file, akin to a README.txt or license.txt
11	class FileRenameService	1✔
12	# See this reference for the full list of characters that cannot be used in filenames for AWS S3:
13	# https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html
14	# This service will only attempt to fix the most likely problems. For example, we will not try to
15	# handle "ASCII character ranges 00–1F hex (0–31 decimal) and 7F (127 decimal)"
16	# Note that we do not rename for a single space, but we do for two spaces together, because according to S3 docs:
17	# "Significant sequences of spaces might be lost in some uses (especially multiple spaces)"
18	ILLEGAL_CHARACTERS = [	1✔
19	"&", "$", "@", "=", ";", ":", "+", " ", ",", "?", "\\", "{", "}", "^", "%", "`", "[", "]", "'", '"', ">", "<", "~", "#", "\|"
20	].freeze
21
22	attr_reader :original_filename	1✔
23
24	def initialize(filename:)	1✔
25	@original_filename = filename	×
26	end
27
28	def needs_rename?	1✔
29	@needs_rename \|\|= check_if_file_needs_rename	×
30	end
31
32	def check_if_file_needs_rename	1✔
33	ILLEGAL_CHARACTERS.each do \|char\|	×
34	return true if @original_filename.include? char	×
35	end
36	false	×
37	end
38
39	# Replace every instance of an illegal character with an underscore.
40	# Append an index number in parentheses just before the file extension,
41	# so we avoid ever accidentally naming two files identically and causing
42	# one to over-write the other.
43	def new_filename(index)	1✔
44	nf = @original_filename.dup	×
45	ILLEGAL_CHARACTERS.each do \|char\|	×
46	nf.gsub!(char, "_")	×
47	end
48	split = nf.split(".")	×
49	split[-2] = "#{split[-2]}(#{index})"	×
50	split.join(".")	×
51	end
52	end

pulibrary / pdc_describe / 9091a1ae-29be-458c-984a-339d213919c4

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous