Search result rendering can crush the system

Bug #1200770 reported by Mike Rylander on 2013-07-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Evergreen
High
Unassigned
2.3
Undecided
Unassigned
2.4
High
Unassigned

Bug Description

We recently addressed an issue where multiple, identical, broad searches would cause extreme load on the db and cause a cascade of other issues, up to and including a system down state from the user's point of view. This can be triggered by entering a search term in the TPAC and then holding down the Enter key. This was addressed by detecting concurrent, identical searches and only performing the first, causing the remaining searches to wait for and then use the results of the first.

However, once those concurrent searches retrieve their cached results, they all individually render the result page directly from the database via CPU-expensive in-db unAPI calls. This commit uses a very similar technique to add a short-term caching layer (10 seconds) per identical unAPI call to relieve the load of this second wave of load coming from concurrent, identical searches.

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/unapi-spam-reduction

Pasi Kallinen (paxed) wrote :

Perhaps we could also disable the form submit once the form's been submitted? (Although that goes against the "minimal javascript" thing)

Mike Rylander (mrylander) wrote :

That would address the other bug, but only when JS is available (IOW, it closes an accidental DoS hole for "nice" clients, but not the root issue).

Dan Wells (dbw2) wrote :

Works as advertised, pushed back through 2.3. Thanks, Mike!

Changed in evergreen:
status: New → Fix Committed
Ben Shum (bshum) on 2013-11-11
Changed in evergreen:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers