pulibrary / pymarc_dedupe / 447d729c-fa48-403a-a433-ffb3be0b5c85

100%

push

circleci

Use postgres database version for very large data sets (#24)

* Green locally - connect to Postgres DB

- Still need to connect to DB in other environments, including CI
- Need to clear DB between tests
- Need to make sure not to make tons of duplicate records (try to find before creating?)

* Set up DB connection for CircleCI

* Linting fixes

* Ensure that the same records are not re-created

* Try setting up environment using dynaconf

* Fix test DB connection

* Try to fix connection to DB in CI

* Remove pyproject.toml for now

* Checkpoint - green, reads and writes to/from database

* Put blocking in marc_to_db.py

* Try to increase test coverage

* Linting fixes

* Try to increase test coverage

* Test for empty input directory

* Make table creation a class method, work on mapping records to DB

* Use streaming JSON for memory performance

* Linting fixes

* Finish writing to CSV

* Linting, change to cluster_id

* More consistent naming

* Small fixes

* Linting fixes

* Increase test coverage for scoring

* rescue if there is no "a" field in author

* Use threading, override xml_reader for error handling

* Add output of comparison experiment - uses data set from Mark Z

* Setup for db comparison

* Add print statement

* Lint

* Increase coverage, lint

* Increase coverage

* Increase test coverage again

* Try using python orb for easier CI caching

* Fix edition mapping, test coverage

* Add pyproject.toml, ignore cache for ruff

* Formatting

* Re-organize into folders

* Add Goldrush to db & report

* Update data for comparison

* Ensure everything is linted, start adding module comments

* Remove unneeded comments in circleci config

291 of 291 new or added lines in 13 files covered. (100.0%)

842 of 842 relevant lines covered (100.0%)

1.0 hits per line