Use batch methods for multi-row grid actions

Bug #1896285 reported by Jeff Davis on 2020-09-18
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Evergreen
Medium
Unassigned
3.5
Medium
Unassigned

Bug Description

There are a few places in the client where performing an action on multiple rows results in at least one API call per row. It would be better to bundle such actions into a single call to a batch method.

For example:

1. In Item Status, upload a CSV file of 100 barcodes.
2. Select all rows.
3. Actions > Mark as Missing.

This results in 100 separate calls to open-ils.circ.mark_item_missing. You can hit the max open-ils.circ drone count pretty quickly this way. I think a batch method would be more efficient in situations like this.

Changed in evergreen:
importance: Undecided → Wishlist
Jeff Davis (jdavis-sitka) wrote :

Another example:

1. Do a patron search that returns a large number of results (e.g. all patrons where profile = "Users").
2. Select all rows.
3. Add To Bucket.

There will be one open-ils.actor.container.item.create call per row, which can exhaust open-ils.actor drones pretty quickly with a sufficiently large number of rows.

Changed in evergreen:
importance: Wishlist → Medium
Bill Erickson (berick) wrote :

+1 to batch API calls.

A potential stop-gap would be to modify the JS to serialize the API calls instead of blasting them all at once. This can typically be done with only minor JS changes.

Bill Erickson (berick) wrote :
tags: added: pullrequest
Changed in evergreen:
milestone: none → 3.6.2
status: New → Confirmed
Jeff Davis (jdavis-sitka) wrote :

Thanks, Bill. Unfortunately, after applying those branches in a test environment, both my test cases still max out available drones and give "no children available" warnings in the logs.

Bill Erickson (berick) wrote :

Thanks for testing, Jeff. Hmm, I'm seeing different results.

I just re-tested the patron bucket scenario:

I set up my VM with max-children=5 for open-ils.actor. Loaded up a patron search with 30 results. At this point I have 2 actor drones alive. Added all of the patrons to a bucket and the number of actor drones rose by exactly one, stopping at 3, which is what I would have expected, given that each call waits in the previous call to complete.

Can you set up a similar test and see now many new actor drones are created in this scenario?

Chris Sharp (chrissharp123) wrote :

Seeing trouble in PINES with this call in the activity log:

2020-12-14 09:24:11 brick04-head gateway: [ACT:102767:osrf-websocket-stdio.c:559:16079558091027671352] [127.0.0.1] [] open-ils.actor open-ils.actor.ou_setting.ancestor_default.batch "188", ["cat.default_copy_status_normal"], "<REDACTED>"

In our case, it caused two separate system outage events in one day.

Jason Stephenson (jstephenson) wrote :

The patch for patron bucket add from comment #3 is working for me on a test VM running 3.2.10. I assume it will work on any later release, too. I'll refrain from adding a sign off branch because Chris Sharp mentioned in IRC that he's also looking at these patches.

If anyone wants to add my signoff to that one, I consent to signing off on the branch with my name and email address: Jason Stephenson <email address hidden>.

Chris Sharp (chrissharp123) wrote :

I have been running all three of the above branches in PINES production for the last day and can vouch for their working - both not affecting anything adversely and definitely seeing fewer system calls resulting in crazy drone explosions. I'm interested in what our OP Jeff Davis thinks about where we are on this before submitting a signoff branch?

Jeff Davis (jdavis-sitka) wrote :

Upon re-testing, the patron bucket add patch is working for me too. I must have done something wrong during my initial test. I think we can assume that one is working just fine.

However, the batch mark missing patch still does not work for me. With the patch applied, when I go through the test steps from the original bug report, I still end up with maxed-out actor drones and "no children available" warnings in logs. It takes a few seconds for the drones to max out -- perhaps the issue is with the post-action grid reload, rather than (or in addition to) the actual action of marking the items as missing? I'm attaching a CSV file with my test set of 100 barcodes to be marked as missing.

Bill Erickson (berick) wrote :

Thanks, Jeff. I'm looking closer at the Missing use case.

Changed in evergreen:
assignee: nobody → Bill Erickson (berick)
Bill Erickson (berick) wrote :

Your instincts were correct, Jeff. The post-action copy loading was running all in parallel. I've pushed another commit to the same mark-missing branch to address this. I also tweaked it so it waits until all of the data has been fetched before it refresh the grid to avoid the grid flickering.

Changed in evergreen:
assignee: Bill Erickson (berick) → nobody
Jeff Davis (jdavis-sitka) wrote :

Thanks Bill, that does the trick for me -- with both commits I no longer see a high drone count when marking many items as missing.

Working branch user/jeffdavis/lp1896285-batch-mark-missing has a signoff for the two commits, plus an additional commit adding a bare progress dialog (since the grid can take a little while to reload).

Jason Stephenson (jstephenson) wrote :

I removed the 3.4 target on this bug because 3.4.6 is going to be a security release, and I don't think this is a security bug.

no longer affects: evergreen/3.4
Bill Erickson (berick) on 2021-01-11
Changed in evergreen:
assignee: nobody → Bill Erickson (berick)
Bill Erickson (berick) wrote :

Thanks Chris and Jeff.

Code merged to master and 3.6

I had a merge conflict in 3.5 that I was able to fix but unable to test, so I have pushed a separate branch for 3.5 in the hopes someone with 3.5 handy can give it a once over. Specifically, this commit resulted in the conflict:

https://git.evergreen-ils.org/?p=Evergreen.git;a=commitdiff;h=0a7eef2a73ddf43dd6dfd313980eb56419832415

3.5 Branch:

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp1896285-batch-grid-actions-serialize-combo-3.5

Changed in evergreen:
status: Confirmed → Fix Committed
assignee: Bill Erickson (berick) → nobody
Jeff Davis (jdavis-sitka) wrote :

We've had the 3.5 branch in production for about a week. Since then we haven't had any problems that I can attribute to the user actions/UIs addressed here, although we've had a couple of instances of max drones that I believe are due to other causes. I think the fixes can be committed to rel_3_5.

See also bug 1912834 (limiting the number of parallel requests from the client).

Blake GH (bmagic) wrote :

We've had the 3.5 patch running for 2 days without issue. My signoff

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/blake/LP1896285_3_5

tags: added: parallel-requests
Jeff Davis (jdavis-sitka) wrote :

3.5 version pushed to rel_3_5.

Changed in evergreen:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments