87b48f38-da87-49b9-8cf4-216ba7d6372e

Committed 06 Jan 2025 06:35PM UTC coverage: 92.014% (+0.06%) from 91.952%

Build # 87b48f38-da87-49b9-8cf4-216ba7d6372e

Build Type

push

circleci

Committed by

web-flow

Commit Message

Refactor unzipping and xml file cleanup (#2586)

- Move tests to overall Job spec to prep for refactor
- Put CSV Validation into its own child job, run download & process in callback
- Refactor unzipping and xml file cleanup
- Rescue from Zip::DestinationFileExistsError
  - If the file already exists, that's fine, just move to the next - may have run part of the unzipping previously
- Respond to review - remove duplicate method

Run Details

52 of 52 new or added lines in 3 files covered. (100.0%)

2 existing lines in 1 file now uncovered.

3422 of 3719 relevant lines covered (92.01%)

374.1 hits per line

Source File
Press 'n' to go to next uncovered line, 'b' for previous

92.0

/app/models/scsb/partner_updates/full.rb

module Scsb
  class PartnerUpdates
    class Full < Update
      def initialize(dump:, dump_file_type:, timestamp: DateTime.now.to_time)
        super
      end

      def self.validate_csv(dump_id, file, institution)
        matches_expected_collections = false
        raise StandardError, "No metadata files found matching #{institution}" unless file

        csv = CSV.read(file, headers: true)
        group_ids = csv['Collection Group Id(s)'].first
        matches_expected_collections = group_ids == '1*2*5*6'
        filename = File.basename(file)
        scsb_file_dir = ENV['SCSB_FILE_DIR']
        destination_filepath = "#{scsb_file_dir}/#{filename}"
        FileUtils.move(file, destination_filepath)
        Dump.attach_dump_file(dump_id, destination_filepath, :recap_records_full_metadata)
        File.unlink(destination_filepath) if File.exist?(destination_filepath)
        unless matches_expected_collections
          raise StandardError, "Metadata file indicates that dump for #{institution} does not include the correct Group IDs, not processing. Group ids: #{group_ids}"
        end

        matches_expected_collections
      end

      def download_full_file(file_filter)
        prefix = ENV['SCSB_S3_PARTNER_FULLS'] || 'data-exports/PUL/MARCXml/Full'
        @s3_bucket.download_recent(prefix:, output_directory: @update_directory, file_filter:)
      end

      def self.download_full_file(institution, extension)
        Scsb::S3Bucket.partner_transfer_client.download_recent(
          prefix: ENV['SCSB_S3_PARTNER_FULLS'] || 'data-exports/PUL/MARCXml/Full',
          output_directory: ENV['SCSB_PARTNER_UPDATE_DIRECTORY'] || '/tmp/updates',
          file_filter: /#{institution}.*\.#{extension}/
        )
      end
    end
  end
end

1	module Scsb	1✔
2	class PartnerUpdates	1✔
3	class Full < Update	1✔
4	def initialize(dump:, dump_file_type:, timestamp: DateTime.now.to_time)	1✔
5	super	3✔
6	end
7
8	def self.validate_csv(dump_id, file, institution)	1✔
9	matches_expected_collections = false	20✔
10	raise StandardError, "No metadata files found matching #{institution}" unless file	20✔
11
12	csv = CSV.read(file, headers: true)	18✔
13	group_ids = csv['Collection Group Id(s)'].first	18✔
14	matches_expected_collections = group_ids == '125*6'	18✔
15	filename = File.basename(file)	18✔
16	scsb_file_dir = ENV['SCSB_FILE_DIR']	18✔
17	destination_filepath = "#{scsb_file_dir}/#{filename}"	18✔
18	FileUtils.move(file, destination_filepath)	18✔
19	Dump.attach_dump_file(dump_id, destination_filepath, :recap_records_full_metadata)	18✔
20	File.unlink(destination_filepath) if File.exist?(destination_filepath)	18✔
21	unless matches_expected_collections	18✔
22	raise StandardError, "Metadata file indicates that dump for #{institution} does not include the correct Group IDs, not processing. Group ids: #{group_ids}"	1✔
23	end
24
25	matches_expected_collections	17✔
26	end
27
28	def download_full_file(file_filter)	1✔
UNCOV 29	prefix = ENV['SCSB_S3_PARTNER_FULLS'] \|\| 'data-exports/PUL/MARCXml/Full'	×
UNCOV 30	@s3_bucket.download_recent(prefix:, output_directory: @update_directory, file_filter:)	×
31	end
32
33	def self.download_full_file(institution, extension)	1✔
34	Scsb::S3Bucket.partner_transfer_client.download_recent(	33✔
35	prefix: ENV['SCSB_S3_PARTNER_FULLS'] \|\| 'data-exports/PUL/MARCXml/Full',
36	output_directory: ENV['SCSB_PARTNER_UPDATE_DIRECTORY'] \|\| '/tmp/updates',
37	file_filter: /#{institution}.*\.#{extension}/
38	)
39	end
40	end
41	end
42	end

pulibrary / bibdata / 87b48f38-da87-49b9-8cf4-216ba7d6372e

Source File Press 'n' to go to next uncovered line, 'b' for previous

Source File
Press 'n' to go to next uncovered line, 'b' for previous