Add option to rebuild reporter.materialized_simple_record to pingest.pl

[Jason Stephenson]

Recent IRC discussion (http://irc.evergreen-ils.org/evergreen/2019-01-11#i_390371) revealed that pingest.pl will not rebuild the reporter.materialized_simple_record table.

As the original author of pingest.pl, I think that rebuilding this table should be added as an option to pingest.pl. This would be a positive option, i.e. to make it rebuild the table, rather than a negative, or skip, option.

[Jason Stephenson (jstephenson)]

So, I have added an option to pingest.pl to update the rmsr entries for the ingested records rather than rebuild the entire table. I decided to go this route because rebuilding the entire table cannot be done in parallel and doing so in pingest.pl would end up being done in its own process and would just do what can be done now, via psql. A feature to completely rebuild the table could still be added, as part of my branch if so desired.

Now that I have done the work and started testing it, this feature seems less useful to me than it did ten days ago in IRC.

Anwyay, here is the branch: http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dyrcona/lp1811696-pingest-update-rmsr-option

I will refrain from adding the pullrequest tag until after my current test run completes successfully.

[Jason Stephenson (jstephenson)]

Here's the branch that actually addresses this bug's description, in retrospect, this option is more useful, so when testing this feature, use this branch and not the one in comment #1.

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dyrcona/lp1811696-pingest-rebuild-rmsr-option

Testing should be pretty simple: Run pingest.pl with the --rebuild-rmsr option. Depending on the size of your database and other options, the first process to finish should be the one that did the rebuild. Check that your reporter.materialized_simple_record makes sense for your version of Evergreen.

A better way to test would be to start out with a database where reporter.materialized_simple_record has the title fields in normalized (i.e. lower case) form. Then upgrade it to a version of Evergreen where the titles should be in mixed case, i.e. 3.2. Now, run pingest.pl with the above option and see that the case of titles has changed.

[Jason Stephenson (jstephenson)]

FYI: I rebased the branch on master today and squashed in a small bug fix for keeping track of the child process that is rebuilding the reporter.materialized_simple_record table.

[Bill Erickson (berick)]

Code works as expected, but I have a question.

Given that reporter.materialized_simple_record is ultimately rebuilt from data in metabib.wide_display_entry (via reporter.old_super_simple_record), should pingest.pl wait to call rmsr_rebuild() until after the main reingest completes and the display entries have been updated?

[Jason Stephenson (jstephenson)]

Removing pull request tag to make the change suggested by Bill Erickson in comment #4.

[Jason Stephenson (jstephenson)]

I force pushed a rebased branch with Bill's suggestion to do the rebuild of the reporter.materialized_simple_record table after the other ingests have finished.

working/user/dyrcona/lp1811696-pingest-rebuild-rmsr-option

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dyrcona/lp1811696-pingest-rebuild-rmsr-option

[Galen Charlton (gmc)]

Pushed to master for inclusion in 3.4. Thanks, Jason and Bill!