build/tools/relator_map no longer fetching relator codes

[Galen Charlton]

As of today, build/tools/relator_map no longer works because the LC website appears to be blocking requests from LWP/Simple user agents. This could be gotten around by changing the user agent (for example, curl is not blocked), but it might also be good to see if nowadays we're better off grabbing the list of relators from id.loc.gov.

Evergreen master

[Galen Charlton (gmc)]

Note that I've verified for the purpose of building 2.12-beta that there have been no relator code changes since the last time Evergreen's copy was updated in 2015.

[Dan Wells (dbw2)]

I noticed this also during the 2.11 cycle, and reached the same conclusions, so marking as confirmed.

[Chris Sharp (chrissharp123)]

Taking a look at this. Initially planning to fix the current approach of downloading the HTML file and using a parser like HTML::TableExtract to grab the codes.

[Dan Scott (denials)]

To get this working in the short term, I pushed a two-line fix that adds an "Evergreen/3.1" user agent to the script to http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbs/lp1666987_build_relator_map_ua

Giving it a local run suggests that no terms have changed since 2015. But it would be nice to have a working script for the build process.

Longer term, we should evolve towards supporting the full URIs in relators.tt2 anyway, as well as the short relator codes, so I would eventually recommend reimplementing this to use the linked open data access method as suggested by Galen... something like:

1. Grab the complete list of relator codes from id.loc.gov (using cURL here as a handy way of dereferencing the URIs with the appropriate Accept: header but should be easy enough to accomplish with LWP::Simple, as id.loc.gov does not use the Cloudflare blocking that the loc.gov main site is):

curl -L -H 'Accept: application/json' http://id.loc.gov/vocabulary/relators/

2. Iterate over the members of "http://www.w3.org/2004/02/skos/core#hasTopConcept", which will give you:

            {
                "@id": "http://id.loc.gov/vocabulary/relators/abr"
            },
            {
                "@id": "http://id.loc.gov/vocabulary/relators/act"
            },
            ...

3. Dereference each of the relator URIs to extract the corresponding label and abbreviated code:

curl -L -H 'Accept: application/json' http://id.loc.gov/vocabulary/relators/abr

        "http://www.w3.org/2004/02/skos/core#prefLabel": [
            {
                "@language": "en",
                "@value": "Abridger"
            }
        ],
        "http://www.w3.org/2004/02/skos/core#notation": [
            {
                "@value": "abr"
            }
        ],

4. Generate our relators.tt2 as before, but make it smart enough to equate the short relator codes with the full URIs (perhaps in the authors.tt2 code)

But in the short term let's just test and merge http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbs/lp1666987_build_relator_map_ua and move forward.

[Jason Stephenson (jstephenson)]

Seeing as we typically only do this during the .0 release, should this be targeted at 3.1 and 3.2 and not 3.next? I understand that some could run this script on their own if they're doing an installation, but doing so is not part of the typical install steps, nor is it documented anywhere other than the release steps as far as I know.

[Ben Shum (bshum)]

I agree with Jason's comment and recommend that we only target this towards the next future major series. It's not typically checked during maintenance releases as far as I know.

[Dan Wells (dbw2)]

Just dealt with this again... for the last time!

Works for me, so pushed to master. Thanks, Dan!