Web Client: Download Block List causes unresponsive page with large file

[Chris Sharp]

In the web client's Offline Circulation interface, clicking the Download block list button locks up the screen when the file is large. The PINES offline block list is 27MB. Generating an offline file on stock data (4 barcodes) works successfully. Not sure where the breakpoint is.

Evergreen 3.0.0 through current master
OpenSRF 3.0.0
PostgreSQL 9.4
Ubuntu 16.04 LTS

[Terran McCanna (tmccanna)]

We've spent a lot of time getting our libraries to use the standalone, but without the offline block list, the benefits of using it are greatly diminished. Although our offline file is very large, it does download successfully (albeit slowly) in the xul client, but it will not download in the web client.

[Blake GH (bmagic)]

Confirmed - 5.7MB file makes the web client go unresponsive. I tried 196K and it worked but still took about 60 seconds. Presumably if we kept clicking on the Chrome dialog for "Wait/Cancel" allowing it to continue processing for a few hours, it would download and process the whole file. But who wants do ask their staff to do that?

[Chris Sharp (chrissharp123)]

I just dug into this a bit more. The file downloads as quickly as the connection allows - on the 25Mb/s connection I'm on this is a matter of seconds. The part that is taking so long for larger files ("larger" probably meaning "more than the few you'd get on a sparsely-populated test dataset") is the "egLovefield" part, which parses each line of the file, then loads it into an offline database. I'm not sure what the alternatives are, but this clearly isn't working for real life data. :-/

[Chris Sharp (chrissharp123)]

Relevant IRC discussion: http://irc.evergreen-ils.org/evergreen/2017-11-28#i_335769

[Chris Sharp (chrissharp123)]

Since it looks like this is not likely to be fixed anytime soon, we have hidden the button locally:

http://git.evergreen-ils.org/?p=evergreen/pines.git;a=commit;h=21db8b772c26dcc12dee4f009fe442a446c0ecc5

At this point, even narrowing the list down to only barred accounts results in a 413KB file that will not load within five minutes (as Blake says, the browser will keep asking the user if they want to wait). In more than one test, the entire workstation froze, requiring a reboot. I'm wondering if we shouldn't remove the button project-wide until this can be redeveloped?

[Jeff Davis (jdavis-sitka)]

We're currently doing one insert for each line of the block list, which isn't very efficient. But even if we insert multiple rows at once, it sounds like we'll run into problems with large lists. From section 2.3 of the Lovefield design doc:

"If the database has more than 10K rows in a single table, users need to consider using experimental bundled mode to improve app loading time."

That 10K row limit will be insufficient in many cases. (Sitka's block list currently has 564K entries!)

Bundled mode is described in section 2.3.2:
https://github.com/google/lovefield/blob/master/docs/dd/02_data_store.md#232-bundled-mode-experiment

Instructions for enabling bundled mode are described elsewhere:
https://github.com/google/lovefield/blob/master/docs/spec/99_postfix.md

Something to explore?

[Chris Sharp (chrissharp123)]

That's a really interesting idea. I read through the documentation on this and found that implementing bundled mode might require some redesign of the current code. Bundled mode appears to only be available when defining the schema statically (https://github.com/google/lovefield/blob/master/docs/spec/07_spac.md) in a YAML file that gets processed into automatically generated code. Right now we generate the schema dynamically (https://github.com/google/lovefield/blob/master/docs/spec/01_schema.md).

[Bill Erickson (berick)]

Here's a branch that speeds it up by batching the barcode inserts into a single DB call.

Note I also tried this:

osb.setPragma({enableBundledMode: true});

It had no noticeable affect on the speed, so I disabled it.

On my test system, I loaded 611k rows in about 50 seconds. In prior tests I gave up after about 10 minutes.

The code contains extra debug logs we might want to remove for merging.

[Bill Erickson (berick)]

The aforementioned branch:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp1727557-offline-block-list-load-speed

[Bill Erickson (berick)]

Also curious if a progress dialog would also be helpful here. It wouldn't show actual progress, but it would make it more clear something is happening.

[Terran McCanna (tmccanna)]

Thanks Bill! We will try to get this tested against our patron data set.

+1 to a progress dialog

[Bill Erickson (berick)]

I have force-pushed back to the same branch, it now includes a progress dialog.

[Jason Boyer (jboyer)]

I do think a progress bar would be good, especially since we'd also fall in the ~1 minute range. If there's no feedback staff will assume something has broken.

[Bill Erickson (berick)]

As a heads up, I've noticed when my offline DB is loaded with ~600k rows it adds a noticeable delay to page rendering on every page. Curious if others can confirm.

[Mike Rylander (mrylander)]

Thanks, Bill! Re delay, how much and does it show up in the browser dev tools timing utilities?

[Bill Erickson (berick)]

Mike, with a full DB audit shows 3.4 seconds of "scripting". After deleting the database, that value drops to .33 seconds of "scripting".

[Cesar V (cesardv)]

Alright, so I've tested Bill's patch and works, there's a definite improvement. For a large block-list (I tested with the attached ~500K list, 6.7Mb) and it still took a minute... like Chris noted. I clocked like 45-50 seconds to ingest. No idea how common having a block list of half a million is though... but it's definitely better with the progress dialog "feedback"! ;)

I too noted the page/tab in Chrome became noticeably slow when IndexDB was loaded with the 500K records... like just going to the webstaff Home page could take like 5-8 seconds. I had to clear the BlockList table in IndexDB AND close the tab, and reopen a new one, for the webstaff to return to normal responsiveness.

[Bill Erickson (berick)]

I've confirmed the delay happens during osb.createTable('OfflineBlocks') in lovefield.js which is called on every page. I'm going to create a patch that allows the browser to ignore the table until it needs it. That should limit the delay to just the offline interfaces.

[Bill Erickson (berick)]

Patch pushed to the same branch (also rebased to master) that segregates the offline block list and transaction data into its own database, which is only initialized in the offline interface. This allows the browser to ignore those large data blobs on most pages.

Note this code change assumes any outstanding offline data is uploaded and processed before upgrading. Ideally, users would delete the offline database first, but it's not strictly required.

[Bill Erickson (berick)]

Fixes for this bug will be incorporated into bug #1775719. Will mark as dupe when we have something to test.

[Jeff Davis (jdavis-sitka)]

I'm reopening this bug because it turns out the issue is not actually fixed. Even with the fix from bug 1775719, Download Block List still causes the page to hang/crashes my browser with a 14MB block list on both 3.1.7 and 3.3.3. Someone else reported the same thing with 3.0.11 (comment #27 on bug 1775719).

[Terran McCanna (tmccanna)]

+1 We had to hide the button in our implementation because no matter how much we restricted our block list, it was still too big and wouldn't download.