OneClickdigital API integration

[Jeff Davis]

OneClickdigital now provides APIs to allow third-party applications to interact with their products and services (see http://developer.oneclickdigital.us/). I am working on adding some functionality to Evergreen to make use of their APIs. Here's the plan:

1. Add a simple, generic HTTPClient utility to handle HTTP requests/responses from third-party servers. (This can also be used by open-ils.resolver and other EG services.)

2. Develop a generic third-party-API integration service for EG, open-ils.ebook_api, which can be used "out of the box" with OneClickdigital.

3. Eventually, add OverDrive API integration using the open-ils.ebook_api service. This will supersede the existing OverDrive integration layer described in bug 1410537, which currently works but has limitations that make it hard to extend for use with non-OverDrive services.

My target for steps 1 and 2 is late summer/fall 2016, with step 3 to follow at some point afterwards.

[Jeff Davis (jdavis-sitka)]

For anyone interested, my (very incomplete, pre-alpha) working branch is here:
http://git.sitka.bclibraries.ca/gitweb/?p=sitka/evergreen.git;a=shortlog;h=refs/heads/user/jeffdavis/ebook-api-upstream

The backend design is somewhat similar to Evergreen's added content module. Common functionality and API calls are defined in the main Perl module, OpenILS::Application::EbookAPI, while vendor-specific details like endpoint URLs are broken out into separate "handler" submodules for each vendor API. The actual mechanics of HTTP requests/responses are handled by the new OpenILS::Utils::HTTPClient module.

The frontend is essentially a JS layer over top of the OPAC, with some light use of Dojo since we're already using it, plus a few additions to TT2 templates. The JS layer uses the OpenSRF translator to talk to the open-ils.ebook_api service, which in turn makes the appropriate calls to the third-party API. (This is a little different from the existing OverDrive integration layer, where the client did most of the work and the server was merely a proxy.) The open-ils.ebook_api service is session-based; when the client initializes a session, connection info is pulled from org unit settings and stored in memcached, and the session ID is saved in a browser cookie. Among other things, this prevents authentication tokens from being exposed to the client.

The user will see the following changes:

- The "Place Hold" link on search results and record summary, which is currently suppressed on e-books, will be replaced with a "Check Out" link, allowing the user to directly check out the title from OneClickdigital (OPAC login required, of course).

- If the title is not available, the "Check Out" link will instead be a "Place Hold" link, allowing the user to place a hold on the title with OneClickdigital.

- For OverDrive, e-book holdings information (number of "copies" available per format) will be displayed in item details. OneClickdigital doesn't have this concept of holdings, so there is nothing to display.

- When the user is logged in, the dashboard will show a count of e-book checkouts and holds. This will be separate from the "main" checkouts/holds display, since checkouts/holds on titles from third-party vendors are unrelated to checkouts/holds in Evergreen.

- Additional tabs in My Account will allow the user to view their e-book checkouts, checkin or renew titles, view or cancel holds, etc.

[Jeff Davis (jdavis-sitka)]

For authenticated users, we want to avoid constantly looking up information about their account (patron ID and existing transactions, including title metadata). We should retrieve that information once on login, cache it, and update the cache when anything changes. I think the best approach here is to do what we do with ebook_api sessions: store the user info in memcached, and have a few basic Perl API calls for retrieving or updating it. The JS layer can use the Perl API calls as needed. We can also use the cache when rendering TPAC pages, particularly the e-book transaction totals in the dashboard, which rarely change. When the patron performs a transaction, the JS layer does the transaction via the ebook_api service, updates the user cache accordingly, and updates the current page if needed.

[Jeff Davis (jdavis-sitka)]

OneClickdigital doesn't do real patron authentication (you just need a valid barcode). However, OverDrive does do auth if configured to do so, so ebook API integration should be able to handle it.

There are two ways to do patron auth via the OverDrive API:

1. Patron Authentication - include the (unencrypted) user password in an API request: http://developer.overdrive.com/apis/patron-auth
2. Granted Authorization - 3-legged OAuth; password is not exposed: http://developer.overdrive.com/granted-auth

Obviously option #2 is preferable. But we don't want to have a separate login page just for the OverDrive API. Since all actual interaction with the external API is handled by the Evergreen backend (the user's browser never talks to OverDrive directly), I am thinking of this approach:

1. When the login page is loaded, the Evergreen backend sends a "granted auth" request to the OverDrive API.
2. OverDrive responds with a 302 redirect to a "dummy" URI (specified by the library when it asks OverDrive to enable granted auth). This response includes an authorization code.
3. Evergreen ignores the redirect, but grabs the authorization code and stashes it in our EbookAPI session cache.
4. The user actually logs in to EG by clicking the "Log in" button. (We could include a checked-by-default "Also authenticate me with OverDrive" checkbox on the login page if we think explicit consent is required here.)
5. Once the user is logged in (i.e. we have an EG authtoken), we pull the OverDrive authorization code from the cache and use it to request an access token from OverDrive.
6. OverDrive responds with the access token, and with a refresh token that can be used to re-auth the user if their access token expires. These are cached by EG and used in subsequent API requests.

For OneClickdigital, we skip steps 1-4. At step 5, we look up the barcode of the logged-in user's active card and use it to request a patron ID from the OneClickdigital API. We cache the patron ID and use it as needed in subsequent API requests.

This feels a bit like cheating in the OverDrive scenario, since we don't actually redirect the patron to a separate auth page, but I think it's a reasonable approach under the circumstances.

[Jeff Davis (jdavis-sitka)]

I should add that this approach presumes that you are importing MARC records from ebook vendors into Evergreen. Both OneClickdigital and OverDrive allow you to search their collections via their APIs, but integrating the API search results into EG search results is not feasible without big changes to how search and display works in Evergreen.

Libraries will also need to request API access directly from the vendors before enabling integration in EG, since you need access tokens and so on to be able to use the API. I'll include documentation for that when the code is ready for proper testing.

[Kathy Lussier (klussier)]

I don't have much feedback on the backend details, but I don't see a need for an explicit checkbox to log into OD at the same time. Even if it's checked by default, I think the additional option just give users one more thing they need to think about.

[Jeff Davis (jdavis-sitka)]

I've pushed a branch to the working repo for review:

user/jeffdavis/lp1541559-ebook-api

This adds the new open-ils.ebook_api service with handlers for the OverDrive and OneClickdigital APIs, and integrates title and patron transaction info from those APIs into the TPAC. The UI does not yet have support for performing actions (checking out or renewing titles; placing, suspending, or canceling holds). I'm hoping we can get the present functionality into the 2.12 release, and add the missing transaction functions in the release after 2.12.

A few minor tweaks are still required (e.g. better-looking display of transaction details in My Account), but this code is ready for review. Documentation is forthcoming as soon as I can write it.

As noted above, the underlying assumption here is that you are loading vendor MARC records directly into Evergreen. It is assumed that these records contained vendor-specific URIs in the 856 field, or in the case of OverDrive, that vendor ebook records contain an appropriate 037 tag (see the changes to parts/misc_util.tt2 for details).

MINIMAL SETUP INSTRUCTIONS:

1. Add the required <http_client> block to your opensrf.xml (see changes to opensrf.xml.example in this branch).

2. Add the open-ils.ebook_api service to opensrf.xml and opensrf_core.xml as required (see examples in this branch).

3. Edit the following settings in parts/config.tt2:

ebook_api.enabled -> true
ebook_api.ebook_test.enabled -> true (optional for production, but required if you want to use the Test handler as described below)

USING THE TEST HANDLER:

As written, the code includes a "Test" handler which can be used to test the new service; live tests use this handler. I've also added a few new MARC records to the concerto dataset that work with the Test handler (see Open-ILS/tests/datasets/sql/bibs_ebook_api.sql); a search for "tolkien" will show those records in the OPAC, and if ebook API is enabled (see below), holdings info will be displayed for those titles. Finally, the Test module displays transaction info for the user with barcode 99999359616; if you login with barcode 99999359616 / password andreac1234, you will see ebook transaction totals in the dashboard and detailed ebook transaction info in My Account.

ADDITIONAL SETUP INSTRUCTIONS:

If you want to set this up to work with the OverDrive and/or OneClickdigital APIs, you will need to do the following:

1. Add the necessary values for the new org settings for those vendor APIs (basic token, library ID, etc.). You will need to contact the vendor to get those values. I'll document the settings more thoroughly in future, but in the meantime, consult vendor API documentation for more details:

http://developer.overdrive.com/
http://developer.oneclickdigital.us/

2. In the TPAC, edit parts/config.tt2. For OneClickdigital, for example, you'd want to adjust the following settings:

ebook_api.enabled -> true
ebook_api.oneclickdigital.enabled -> true
ebook_api.oneclickdigital.base_uris -> [ comma-separated list of URI pattern(s) you use in the 856 field of your OneClickdigital MARC records ]

(NB: OverDrive Granted Authentication with 3-legged OAuth, previously discussed in thi...

[Kathy Lussier (klussier)]

There must be too much merging going on. I'm getting a merge conflict on this branch already. Jeff, can you rebase your branch against current master? Thanks!

[Jeff Davis (jdavis-sitka)]

Try this branch:
http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/jeffdavis/lp1541559-ebook-api-master

[Jeff Davis (jdavis-sitka)]

Pushed a few bugfixes to the latest branch.

[Kathy Lussier (klussier)]

Hi Jeff,

I tried using this branch to hook up a test server to Overdrive, but I've reached a stumbling block where I'm getting a 500 response from the Overdrive server. According to the Overdrive docs, this indicates there is a problem on their end, though I suspect I may have entered something incorrectly.

In the meantime, though, I noticed a couple of things:

In my logs, I see a request going to http://api.overdrive.com/v1/libraries/[libid]

where my library's id is in place of [libid] in the URI. However, in the Overdrive docs, the uri they provide uses https . Does that request need to be sent via https? I did make that change locally, and it didn't fix my specific problem, but I wanted to check to see if it needs to be changed.

I also just wanted to mention that I'm unable to import the test records into the database. When I try to run the sql, I get the following message:

psql:bibs_ebook_api.sql:8: ERROR: relation "marcxml_import" does not exist
LINE 1: INSERT INTO marcxml_import (tag, marc) VALUES

It's not a big deal, but should probably be addressed at some point.

Thanks!

[Jeff Davis (jdavis-sitka)]

You're getting that SQL error because bibs_ebook_api.sql is not intended to be run directly. It's supposed to be called by load_all.sql, which creates the marcxml_import table and sets a value for the :bib_tag variable. The best way to get the test MARC records into your system is to use the eg_db_config Perl script with the --load-all option.

As for using HTTPS, yes, by default it will use plain old unencrypted HTTP. You should be able to enforce HTTPS by setting the ebook_api.overdrive.discovery_base_uri org setting to "https://api.overdrive.com/v1", and making a similar change for the ebook_api.overdrive.circulation_base_uri setting. If HTTPS doesn't work, you may need to adjust SSL settings in the <http_client> block in opensrf.xml (this is the configuration for the new OpenILS::Utils::HTTPClient module, which manages HTTP requests for the open-ils.ebook_api service). I found it cumbersome to get those settings right in some environments, which is why it defaults to HTTP currently, but I agree that using HTTPS out of the box would be preferable.

[Kathy Lussier (klussier)]

Oh, ha ha. I probably already had the test records loaded, then.

Thanks for the info on https Jeff!

As I already reported in IRC, I've cleared my stumbling block, and things look good so far. I'll be looking at a few more things before hopefully signing off on the code.

[Galen Charlton (gmc)]

> but I agree that using HTTPS out of the box would be preferable.

FWIW, I feel more than a bit strongly that we should not be
distributing configurations that default to using HTTP when HTTPS is
available.

[Kathy Lussier (klussier)]

I added my signoff to working and added a commit to make a small CSS tweak and to correct a link on the e-book checkouts page. If I can get a signoff on the top commit, I can merge the branch for inclusion in 2.12.

[Kathy Lussier (klussier)]

Forgot to post the branch. http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/kmlussier/lp1541559-ebook-api-master

[Mike Rylander (mrylander)]

Jeff, can you detail the problems you had getting HTTPS to work, and what environment was giving you problems? Since this API sends patron details out from the server over the wire, I think we should see what it'll take to complete the security work.

Thanks!

[Jeff Davis (jdavis-sitka)]

Pushed a couple of commits -- (1) a signoff for Kathy's display tweaks, (2) defaulting to HTTPS for OverDrive requests:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/jeffdavis/lp1541559-ebook-api-master

Note that OverDrive's responses may contain non-HTTPS URIs; we don't use these embedded URIs, but they may show up in your logs. Also, HTTPS is hard-coded for some authentication-related OverDrive API endpoints which don't appear to support plain HTTP at all.

***

Mike, the trouble I had was with getting HTTPClient to validate certs properly. HTTPClient is a wrapper around LWP::UserAgent, which uses a few different options for certificate validation -- see the documentation for the $ua->ssl_opts attribute method here:

http://search.cpan.org/~oalders/libwww-perl-6.19/lib/LWP/UserAgent.pm

I ended up adding a section to opensrf.xml.example to let the user specify values for the various certificate-related variables. By default, certificate validation is enabled and CA certificates are presumed to be installed in a location where LWP::UserAgent can find them automatically. This works for me on Ubuntu 14.04, and it appears to work for Kathy as well. In other environments, it might be necessary to tweak the CA cert path/file variables for cert validation to work.

[Kathy Lussier (klussier)]

Thanks for making those changes Jeff. It works for me! I've merged the code to master for inclusion in 2.12.

A couple of housekeeping matters:

We still need release notes for the new feature. Jeff, would you be able to writing something up? If not, I'll try to add them to some other release note entries I'll be writing over the next couple of days.

We also need to open a new bug for performing actions on e-books (checking out, placing holds, etc.) from the catalog. Here's looking forward to more e-book goodness in 3.0!

Thank you for all your work on this Jeff!

[Ben Shum (bshum)]

We found that the new bibs that were added for test ebook records for the sample dataset broke a series of live tests by shifting the IDs for some copies that get generated.

I've included a really hacked workaround that allowed me to still add the new ebook bibs, but keep the generated copies the same as before.

This is a quick working branch with the workaround. I'd like to get better opinions on how we handle the addition of new bib records without breaking tests though...

user/bshum/move-test-ebook-bibs

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/bshum/move-test-ebook-bibs

[Kathy Lussier (klussier)]

True happiness is seeing the words "All tests successful" appear on one's screen. Moving the e-book bibs to the end made my test happy. Now on to doing the same with metarecord bibs. Thanks Ben!

[Kathy Lussier (klussier)]

Jeff, I created a release notes entry for this feature, but please review them and feel free to adapt / add on where I may have gotten something wrong.

Also adding a note that the build tests are failing for the Overdrive and OneClickdigital perl modules. I've filed a separate bug 1666928 on these failures. I would like to resolve the test failures before the RC release three weeks from now.

[Jeff Davis (jdavis-sitka)]

There ought to be a setting for the OneClickdigital API base URI, to be consistent with OverDrive API support and to make it easy to switch between OneClick's integration and production environments. I've pushed a commit that adds this setting - see working branch user/jeffdavis/lp1541559-oneclickdigital-base-uri.

[Terran McCanna (tmccanna)]

Adding reference to new bug and patch related to Jeff's last comment:

https://bugs.launchpad.net/evergreen/+bug/1668414

[Jeff Davis (jdavis-sitka)]

Release notes pushed to branch working/user/jeffdavis/lp1541559-ebook-api-phase1-release-notes

[Jeff Davis (jdavis-sitka)]

See bug 1673870 for ebook transactions - checkouts, holds, etc.