pulibrary / bibdata

92%

main: 91%

LAST BUILD ON BRANCH passing_marc_overhead

branch: passing_marc_overhead

CHANGE BRANCH

x

Reset

• passing_marc_overhead
• 1198-index-temp-locations-squash
• 1373-submit-collection
• 1633-remove-voyager-alma-mapping
• 1665-changeTheSubject
• 1688-discrete-fields
• 1750-language-field-v2
• 1750-primary-language-field
• 1841_cjk_browse_at_index_time
• 1848-update_fragment_to_viewer_container
• 1852-firestone_pf-circulate
• 1867-qa-webhook
• 1929-dont-allow-duplicate-events
• 1951-index-temp-locations
• 1969-temporary-locations-doc
• 1985-remove-filetoservice-and-unusedfixtures
• 2007-firestone-isc
• 2011-rubocopStyleHashSyntax
• 2022-non-requestable-sc-locations
• 2023-583-dontindex-private-notes
• 2032-773w-774w
• 2055-linked-fields
• 2071-barcode-endpoint-new-locations
• 2076-cgd-committed
• 2077_action_note_display
• 2078-process-scsb
• 2136-cleanup-code
• 2142-update-scsb-availability
• 2148_remove_language_code_s
• 2266-rails7.1
• 2266-upgrade-rails7.1
• 2329-check-index-timestamp
• 2356-delivery-locations-change
• 2414-content-advice
• 2415-bibdata-alma-job-status
• 2426-firestone-pb
• 2443-firestone-flmb
• 2445-firestone-isc
• 2448-tech
• 2451-ruby3.2-upgrade
• 2456-document-missing-events
• 2456-missing-events-staging
• 2458-continuous-deployment
• 2475-rails-7.2
• 2486-bibdata-CDL
• 2500-remove-marquand-ph
• 2524-language-name-display
• 2536-bibdata-sqsl-poller
• 2548-track-honeybadger-history
• 2554-archival-item
• 2557-ruby-3.3.6
• 2577-update-pending-tests
• 2597-engineer-res
• 2601-remove-deliveryLocation-PN-for-recap
• 2602-Lewis-have-only-delivery-PT
• 2603-PPPL-delivery-PT
• 2604-eng-delivery
• 2612-annex-remove-lewis
• 2614-bundler-2.6.3
• 2625-node-22
• 2638-document-ephemera-dspace-fields
• 2639-access_facet-ephemera
• 2639-date_created
• 2639-date_created_comb_publisher-into-pub_date_display
• 2639-description-display-update
• 2639-ephemera-concepts-rename
• 2639-ephemera-coverage
• 2639-ephemera-v11
• 2639-ephemera-v12
• 2639-ephemera-v13
• 2639-ephemera-v18
• 2639-ephemera-v19
• 2639-ephemera-v21
• 2639-ephemera-v7
• 2639-ephemera-v8
• 2639-ephemera-v9
• 2639-map-pamphlets-to-book
• 2639-page_count-description_display
• 2639-pub_citation_display-update
• 2639-sort_title-ephemera
• 2787-invalid-subfield-code
• 2791-theses-rake-task-require
• 2801-commons
• 2803-arch-fac
• 2804-add-marquand-pjm
• 2828-update-coverage-origin-structure
• 2851-dspace-7-theses
• 2888-update-electronic-access-desirialize
• 2888-update-electronic-access-serialize
• 2898-lc-homoit-subjects-ephemera
• 2918-primary-seconday-author_roles_1display
• 2920-thumbnail-dimensions
• 2924-rescue-ephemera-rake
• 2926-pickup-locations-update
• 3-deploys
• 3.4.1-ruby
• 4929-status-ol
• 873_add_852_sub_k
• access_restrictions
• accumulate-per-field
• action_note_docs
• add-a-test
• add-ephemera-genre-list
• add-event-id
• add-flipper
• add-gems
• add-logging
• add_new_boxes
• address-cargo-clippy-warnings
• alma_code_start_22-rust
• anonymize_fixture
• another-benchmark
• api_key
• ark_cache_documentation
• augment_revamped
• augment_separate_lists
• augment_squashed
• augment_subject_indigenous
• augment_subject_set
• axe
• axe-accessibility-tests
• bearer
• bibdata-qa
• biography-bug
• break_down_background_jobs
• break_up_toc
• build_call_number-rust
• bundle-update-03142025
• bundle-update-04042025
• carolyncole-patch-1
• change-name-of-born-digital
• change-the-subject
• change-to-bibdata.lib
• change_and_augment_subject_bug
• christinach-patch-1
• christinach-readme-staging-locations
• ci-bearer-img
• circleci-failure
• cleanup
• cleanup-rubocop.yml
• clippy
• copy-of-ruby-3.1-v2
• copy-with-3.1-include-rare-onsite-locations-in-status
• correct-gemfile-groups
• coverage
• coverage-in-ci
• create-ephemera-electronic-access-mod
• create-solr-document-file
• csp
• debugging_traject_error
• dependabot/bundler/actionpack-7.1.5.1
• dependabot/bundler/activerecord-7.2.2.2
• dependabot/bundler/activestorage-7.2.2.2
• dependabot/bundler/cgi-0.4.2
• dependabot/bundler/net-imap-0.5.6
• dependabot/bundler/net-imap-0.5.7
• dependabot/bundler/nokogiri-1.18.3
• dependabot/bundler/nokogiri-1.18.4
• dependabot/bundler/nokogiri-1.18.8
• dependabot/bundler/nokogiri-1.18.9
• dependabot/bundler/rack-3.1.10
• dependabot/bundler/rack-3.1.12
• dependabot/bundler/rack-3.1.14
• dependabot/bundler/rack-3.1.16
• dependabot/bundler/rack-session-2.1.1
• dependabot/bundler/thor-1.4.0
• dependabot/bundler/webhook_monitor/src/nokogiri-1.18.3
• dependabot/bundler/webhook_monitor/src/nokogiri-1.18.4
• dependabot/bundler/webhook_monitor/src/nokogiri-1.18.8
• dependabot/bundler/webhook_monitor/src/nokogiri-1.18.9
• deploy-bibdata-alma1
• deploy-namespace
• deploy-ruby3
• deploy-ruby3.4
• deploy-target-dir
• deprecation_warnings_in_tests
• deprecations
• digital-content
• do_not_index_any_private_partner_records
• doc-event-empty-dump
• document_augment_workflow
• document_import_partner_full_workflow
• document_spec
• dotenv
• dump-event-id-constraint
• dump_file_type_to_enum
• electronic-access-1display-ephemera
• electronic-access-deserialize
• electronic-accessRS-rename-to-iiif_manifest_paths
• electronic_access_1_display
• electronic_access_1display-ephemera
• enum_for_dump_type
• ephemera-2639-collection
• ephemera-2639-index-task
• ephemera-2639-update-test
• ephemera-2639-v14
• ephemera-2639-v15
• ephemera-2639-v16
• ephemera-2639-v17
• ephemera-2639-v18
• ephemera-2639-v6
• ephemera-format-remove-exact_match-check
• ephemera-format-support-ids
• ephemera-language-nestedId
• ephemera-mod
• ephemera-pub_citation_display_combined
• ephemera-thumbnail
• ephemera-thumbnail-2
• ephemera-thumbnail-2898
• ephemera-v4
• ephemera-v6
• ephemra-rake-prod
• error-handle
• facet-fields-vocabulary
• failing-rust-test
• failing-test-figgy-language-008
• fast-subject-facet
• faster_process_xml_file_job
• figgy-language-three-char
• fix-deploy
• fix-origin-ephemera
• fix-running-marc-to-solr-tests
• fix-update-holding-location
• fix_event_seeds
• fix_solr_in_docs
• flaky-test
• follow-up
• format-enum
• geographic_subject_augmented
• gyoku
• health-monitor-fail
• holding-id-temp-locations
• holdings_management
• homosaurus-terms
• i1796-error-capture
• i1858_augment_the_facet
• i1921_remove_datadog_alerts_in_staging
• i2034_geographic_subject
• i2059-no-fees-in-patron-call
• i2069
• i2069-iso639-3
• i2123-revert-file-changes-in-test
• i2157-dissertation-embargo
• i2161_no_solr_wrapper_in_ci
• i2187_indigenous_languages_facet
• i2231-book-shelf-location
• i2239-sru
• i2290_purge_stale_files
• i2308-876x
• i2313_node_upgrade
• i2321_scsb_import_broken
• i2321_scsb_import_job
• i2321_scsb_import_new
• i2321_scsb_import_new_red
• i2321_scsb_import_rebased
• i2321_use_native_sidekiq_jobs
• i2323_upgrade_sidekiq
• i2328_indexing_updates
• i2383_do_not_index_private_scsb
• i2413-institution-identifier-prefix
• i2414_content_advice_mk
• i2416_alert_when_bibdata_event_has_failed_alma_job
• i2508-primary-sources
• i2515-citation-display-fields
• i2543_background_numismatics
• i2593_flaky_test
• i2618_lc_facet_colons
• i2639-ephemera-v20
• i2647_place_of_pub_facet
• i2678_unspecified_countries
• i2679_delimiter_bug
• i2693_alma_figgy_in_online_facet
• i2719-language-iana-s
• i2720-yjit
• i2780_character_indexing
• i2853-scsb-import-full
• i3476_prod_update
• i3962_ldap_flag
• i4909_iana_codes
• i579_index_skbb
• i85-orangetheses-index-error
• iiif-manifest-ephemera
• in_temporary_location-rust
• include-rare-onsite-locations-in-status
• include-worker-in-whenever-roles
• increase_solr_timeout
• index_manager_quick
• index_portfolio_notes
• indexer-rubocops
• instrumented-recap
• int-years
• invalid-scsb-utf8
• invalid-tags
• is-fast-in-subject-facet
• iso-639-3-in-rust
• iso-639-5-in-rust
• issue_templates
• issues-1260-jrgriffiniii-load-test
• keep-obsolete-zned-zscl
• language-docs
• language-two-letter
• lazy-static
• lc_subject_display
• lc_subject_facet
• lcgft
• library_facet
• light-refactor
• lizgarciao-accessibility-1-1
• lizgarciao-patch-1
• lizgarciao-patch-2
• lizgarciao-patch-5
• lizgarciao-patch-6
• local_subject_headings
• main
• marc-rust-basics
• marcbreaker-field
• mask_headings_without_replacement
• microforms
• mockito
• more-alphabetical-order
• move_full_out_of_event_record
• move_xml_processing_into_background_job
• nested_pg_calls
• new-iso-language-field
• no-circ-delivery-locations
• no-tag
• no-webdrivers
• node16
• non-private-items
• normalize-ephemera-id
• numismatics_documentation
• oclc-into-its-own-module
• oclc_normalize_rust
• oclc_number_rust
• oj
• omniauth
• omniauth-rails_csrf_protection
• one-instance-change-the-subject
• open3
• open_ssh_bug
• orangetheses
• orangetheses-i57-upgrade-gem
• orangetheses-main
• orangetheses-v2
• orangetheses-v3
• orangetheses-v4
• orangetheses-v5
• orangetheses-v6
• orangetheses-v7
• orangetheses-v8
• other-versions-rust
• parallel-publication-statements
• part-of-orangelight-4095
• partner_index_from_scratch
• partof-2639
• patron_barcode
• plasma-physics-open
• playing_with_sets
• plum-ark-cache
• pluralize-primary-source
• pr2225
• pr2225b
• pr2229
• prep_for_more_background_jobs
• prepare_for_more_sub_jobs
• process-holdings-use-is_scsb-rust
• profiling
• protecting-envvars
• provencal-old
• pub_created_display-rust
• put_in_initial_batch
• rails-6-1-7-4
• rails6-passingSQS
• rails6-upgrade-v2
• rails6-v4
• rails_5.2.7
• rails_env
• rake-task-delete-ephemera
• readme_updates
• refactor-enums
• refactor-format-2
• refactor_partner_full_index_incremental
• refactor_process_holdings
• refactor_unzip_process
• regex-once
• remove-PN-from-holdings-delivery
• remove-autoloading
• remove-born-digital-link
• remove-class-method
• remove-deprecation-warning
• remove-duplicate-ephemerfolder
• remove-ezid-stubbing
• remove-notes-ephemera-solr-mapping
• remove-on-site-from-flipper
• remove-redundant-rust-tests
• remove-solr-fix-1198-index-temp-locations
• remove-stale-gem
• remove-unused-fields
• remove-unused-stub
• remove-unused-support
• remove-yard
• remove_deprecated_endpoints
• remove_etas_code
• remove_log_of_fixes
• rename-figgy-env
• replace_solr_wrapper_03
• replace_solr_wrapper_04
• require-false-rest-client
• rescue-message-poll-error
• rescue_from_regex_errors
• rescue_rm
• restruct-item
• restructure-leader-mod
• revert-1909-patron_barcode
• revert-2797-4929-status-ol
• rspec-deprecation
• rubocop-fixes-ci
• rubocop-gems
• ruby-3.1-v2
• ruby-3.4.1
• ruby-version
• ruby344
• run_figgy_cron_on_all_workers
• rust-coverage
• rust-current_location_code
• rust-ephemera-mapping
• rust-ext
• rust-genre
• rust-permanent_location_code
• rust-theses
• rust_date
• sandbergja-patch-1
• scsb-doc-1
• scsb-doc-2
• scsb-doc-3
• scsb-index
• scsb_doc_rust
• scsb_partner_record_request
• scsb_update_investigation
• semgrep
• set-default-url-options
• setup_off_of_vpn
• sidekiq-require
• sidekiq_experiment
• single-valued-pub-date
• smaller-fixture-holding-locations-file
• solr_config
• solr_experiment_index_rake
• speed-up-ci
• sqs_poller_make_dump_in_one_transaction
• status-orangelight-4929
• strip-no-numeric
• support-double-id-structure-ephemera
• support-origin-coverage-country-exactMatchId-nested
• temp_location_cleanup
• temp_location_cleanup_broken
• template-updates
• term-ansicolor
• test-both-terms
• test-different-ruby-path
• test-homoit-genre-facet
• theses-doc
• theses-module
• thesis-doc
• traject
• traject_error
• traject_error_handling
• translation_map
• translation_maps_geo
• translations
• type-alias
• unavailable_aeon_items
• uniform_title_display
• unpin-orantheses-from-main
• unreachable-code-bib-types
• unused-gems
• update-04162025
• update-collection-name
• update-ddtrace
• update-dependencies-jan-22
• update-doc-location-changes
• update-docs-location-changes
• update-eastasian-microfilm-deliver-location
• update-ephemera-project-url
• update-gem
• update-gems
• update-gems-ch
• update-index-rake-task
• update-indexing-doc
• update-location-changes-documentation
• update-location-templates
• update-numismatics-indexer
• update-omniauth
• update-on-2433
• update-orangetheses
• update-rails-6.1
• update-readme
• update-templates
• update-test-config
• update-url-test-webhook
• update_bad_dump_docs
• update_change_subject
• update_orangetheses
• update_pg_and_sidekiq
• update_solr_conf
• update_solr_config
• updates
• updates-02072025
• upgrade-gems-2024-02-05
• upgrade-orangetheses
• upgrade-ruby-3.0.4
• upgrade-ruby-3.1.0
• upgrade_change_the_subj_0_4_2
• upgrade_change_the_subject
• upgrade_honeybadger
• upgrade_jammy
• upgrade_lando_pg
• upgrade_node_jammy
• upgrade_nodejs
• uri_escape_deprecation
• use-datadog-gem
• use-library-standard-numbers
• use_change_the_subject_gem
• use_organization_issue_templates
• v2-1198-index-temp-locations
• voyager
• warnings
• webhook-3.2
• webhook-bundle-update
• webhook-ruby-3.3
• webhook_tests_passing
• what-to-remove-augment
• workers-on-worker-machine

Build # 33aae051-7eae-41c9-8619-43ca8ce2cbe0

Build Type

push

circleci

Committed by sandbergja

Commit Message

Experiment: do some traject processing in Rust

This is not ready for production.  It was an experiment to see how
much overhead it would be to pass a whole MarcXML record to rust
during a traject indexing routine.

It is slower, but not a lot slower -- and memoizing the Ruby method
more than made up for the overhead.

The results:
Original ruby implementation: 45288 records in 185.262 seconds; 244.5 records/second overall.
Passing the entire MarcXML document to rust, and reparsing it there: 45288 records in 194.183 seconds; 233.2 records/second overall.
Passing the entire MarcXML document to rust, and reparsing it there, but memoizing the results in Ruby: 45288 records in 180.621 seconds; 250.7 records/second overall.

Run Details

2 of 2 new or added lines in 1 file covered. (100.0%)

3497 of 3814 relevant lines covered (91.69%)

385.79 hits per line