Flood of open-ils.actor.user.has_work_perm_at.batch requests for VIEW_USER with null authtoken

Bug #1990306 reported by Jeff Davis
18
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Evergreen
Fix Released
High
Unassigned
3.8
Fix Released
High
Unassigned
3.9
Fix Released
High
Unassigned

Bug Description

EG 3.9

We've had several incidents where our servers suddenly start receiving an overwhelming number of these requests from the staff client:

open-ils.actor open-ils.actor.user.has_work_perm_at.batch null, ["VIEW_USER"]

In one case, the volume of requests was large enough to knock out the open-ils.actor service.

Circumstantial evidence suggests that these requests happen when the staff client is left open and the session eventually times out. In one instance, when we saw this happen overnight, the only client activity at the time was a redirect to the login page from an IP that hadn't been active since the previous afternoon.

We didn't experience this issue on EG 3.7. I suspect it was introduced by changes made to Open-ILS/web/js/ui/default/staff/circ/patron/app.js in 3.8 -- note that the resolver for egPatronApp now does a VIEW_USER perm lookup upon startup.

Revision history for this message
Jeff Davis (jdavis-sitka) wrote :

See also bug 1940698, another bug where session timeout can lead to request spamming.

tags: added: parallel-requests
Revision history for this message
Jeff Davis (jdavis-sitka) wrote :

Working branch user/jeffdavis/lp1990306-null-authtoken-perm-lookup tries to avoid doing the perm lookup when there's no authtoken. Not tested yet, so no pullrequest.

Revision history for this message
Jeff Davis (jdavis-sitka) wrote :

We're using this fix now and it doesn't seem to break anything, so I'm adding the pullrequest tag.

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/jeffdavis/lp1990306-null-authtoken-perm-lookup

tags: added: pullrequest
Changed in evergreen:
importance: Undecided → High
Changed in evergreen:
assignee: nobody → Chris Sharp (chrissharp123)
Revision history for this message
Chris Sharp (chrissharp123) wrote :

With the assistance of Jason Stephenson and Bill Erickson, I have added an additional check for an active authtoken a the auth service level, which has solved the problem in PINES production. Here's a branch with my signoff for Jeff's fix and my additional commit:

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/csharp/lp1990306-null-authtoken-perm-lookup

Changed in evergreen:
assignee: Chris Sharp (chrissharp123) → nobody
Revision history for this message
Jane Sandberg (sandbergja) wrote :

Thanks, Jeff and Chris! Signoff for both commits at https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/sandbergja/lp1990306-null-authtoken-perm-lookup -- I made a small correction to add a closing curly brace.

tags: added: signedoff
Revision history for this message
Galen Charlton (gmc) wrote :

Pushed on down to rel_3_8. Thanks, Jeff, Chris, Jane, Bill, and Jason!

Changed in evergreen:
milestone: none → 3.10.1
status: New → Fix Committed
Revision history for this message
Galen Charlton (gmc) wrote :

Apologies, I must have been distracted. I've now pushed to the branches.

Changed in evergreen:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.