Enhanced Concerto dataset (Cont.)

[Blake GH]

Continuing bug 1901932

copy/paste from the last comment:

I promptly edited this code based upon the feedback.

Highlights:

- A date carry forward feature that carry's the various date columns forward based upon the difference between today's date and the create_date for asset.call_number in the dataset. Which is the default. This can be skipped with with: psql -v skip_date_carry='1' -f load_all.sql

- Expansion of special cases for certain tables: config.metabib_class, config.org_unit_setting_type, config.global_flag.

- Dropping these tables from consideration: acq.acq_lineitem_history, acq.acq_purchase_order_history, permission.perm_list

I couldn't think of a good way to deal with the timezone issue. My first thought was to check the column for timezonetz and emit the data onto the disk without the timezone (-06). It seems, though, that the dates land in the database OK. And with the date carry forward business, the result looks sound. The drawback is git churn when the dataset is upgraded through versions of Evergreen. It looks* like the files change because they were restored to your local database, then dumped out again.

On a timezone unrelated note:

Working on this may have revealed an issue with some of our config.org_unit_setting_type rows. The seed (950) doesn't appear to insert some of these rows. Whereas, the upgrade scripts do:
opac.did_you_mean.low_result_threshold
opac.did_you_mean.max_suggestions
search.symspell.keyboard_distance.weight
search.symspell.min_suggestion_use_threshold
search.symspell.pg_trgm.weight
search.symspell.soundex.weight

Therefore, the enhanced concerto set, today, includes them.

I've created a new branch, since this branch was merged already.

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/blake/lp1901932_enhanced_concerto_dataset_tweak

[Galen Charlton (gmc)]

Re those six OU settings, they were deprecated in bug 1997485 and no longer included in new installs, but not fully removed (on the theory that an Evergreen admin may want to refer to their old values before fully committing to the new search class versions of those settings.

Consequently, they don't need to be in Enhanced Concerto, and I'll open a separate bug for their outright removal.

[Galen Charlton (gmc)]

That new bug is bug 2023692

[Chris Sharp (chrissharp123)]

Signoff branch with additional commit removing the file adding the unused settings:

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/csharp/lp1901932_enhanced_concerto_dataset_tweak

[Blake GH (bmagic)]

Rebased my original plus Chris's on top. Force pushed:

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/blake/lp1901932_enhanced_concerto_dataset_tweak

[Jeff Davis (jdavis-sitka)]

I've targeted this to 3.11 and 3.12 on the theory that these improvements are bugfixes for existing code rather than enhancements. I won't object if others disagree with that assessment, though. :)

[Jane Sandberg (sandbergja)]

Thanks for your work on this, Blake and Chris. We reviewed these (with Blake's invaluable assistance) in the collaborative code review sessions. We all love the updated dates in enhanced concerto! Steven and I have signed off on it, and I pushed it to rel_3_11 and above.