Malicious search queries can cause downtime
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Evergreen |
Fix Released
|
Critical
|
Unassigned | ||
3.10 |
Fix Released
|
Critical
|
Unassigned | ||
3.8 |
Fix Released
|
Critical
|
Unassigned | ||
3.9 |
Fix Released
|
Critical
|
Unassigned |
Bug Description
EG 3.1, OpenSRF 3.0.1
This afternoon at Sitka, someone performed the following OPAC search:
bib search: #CD_documentLength #CD_meanHarmonic #CD_uniqueWords core_limit(100000) badge_orgs(1,24,33) estimation_
As an SQL injection attack, it failed -- EG did not execute that stuff as SQL. But the search still consumed all available memory on our database server in under 10 minutes, leading to service failure.
In this particular case, I suspect that nested parentheses in the search query contributed to the problem. In a test environment, the following search became a long-running query that ate up gigabytes of memory after several minutes:
foo(bar(
A comparable search without parens returns zero results almost immediately:
foo bar baz wibble wobble wubble flob
To mitigate future problems, we are now killing long-running search queries more aggressively (previously we were killing them at the 10-minute mark, which was insufficient). But I feel like EG could do more to defend against this type of attack.
Changed in evergreen: | |
milestone: | none → 3.6.2 |
status: | New → Confirmed |
no longer affects: | evergreen/3.4 |
no longer affects: | evergreen/3.5 |
Changed in evergreen: | |
importance: | Undecided → Critical |
Changed in evergreen: | |
assignee: | nobody → Jason Stephenson (jstephenson) |
tags: | added: needsreleasenote signedoff |
Changed in evergreen: | |
milestone: | none → 3.11-beta |
Changed in evergreen: | |
assignee: | nobody → Jason Stephenson (jstephenson) |
Changed in evergreen: | |
assignee: | Jason Stephenson (jstephenson) → nobody |
Changed in evergreen: | |
milestone: | 3.11-beta → 3.11.0 |
Changed in evergreen: | |
milestone: | 3.11.0 → 3.11-beta |
status: | Confirmed → Fix Released |
no longer affects: | evergreen/3.6 |
no longer affects: | evergreen/3.7 |
information type: | Private Security → Public Security |
Hey Jeff, thanks for the details.
There is obviously a problem with the "pullup" step of query parsing. The point of that code is to pull child parse tree nodes' atoms (individual search terms) that use the same search class and joiner into their parent, depth-first, in an effort to make the tree as shallow as possible, which translates into fewer join branches in the generated SQL. Your example should flatten completely, but obviously doesn't.
I'll be looking ASAP, but the more eyes the merrier!