certain queries are hard on the database

[Jason Etheridge]

This is just to throw out some ideas to handle this. Occasionally we see pathological queries that even in small quantities can put great load on a system. What can we do about that?

1) canned results, or even an option for dedicated pages/resources for certain search terms

2) variable cache timeouts for search results. The longer it takes for a query to return results, the longer those results will live in memcached

3) better handling of identical concurrent queries (perhaps by stubbing out searches in memcached even before the results are stored, so that repeat queries will just wait a moment and try memcached again, rather than falling through to the database)

Anything else?

[Jason Etheridge (phasefx)]

Apparently you can hold down the enter key in the TPAC when submitting a search and spam the search query. Bradley's going to open a bug for that one, but I just wanted to mention it here.

[Mike Rylander (mrylander)]

I have a branch at security/lp1172936-spam-search-guard that should protect against this sort of thing based on option (3) above. Eyeballs? Testers? If it works as I intend, it will also address bug 1173008.

[Mike Rylander (mrylander)]

At Galen's recommendation, I've added a backstop to avoid the possibility of an infinite while-loop, were we to get into a situation where either timing conspires to keep a looping process behind other attempts at a search, or our automatic cache expiration fails us. Because the stub timeout is 1/3 of $cache_timeout, the backstop of a full $cache_timeout span seems reasonable. Same bat-repo, same bat-branch.

[Mike Rylander (mrylander)]

I added a final commit to make sure that the "running" indicator in the above branch is removed when appropriate. Testing beyond Jason Etheridge and I would be great.

We're now down to a possible DoS on apache itself, but the database will not get crushed from repeated, identical, concurrent queries any longer. mod_security could help with the apache issue, but that's another bug altogether.

[Jason Etheridge (phasefx)]

I've pushed my sign-off to security/lp1172936-spam-search-guard-jason

[Mike Rylander (mrylander)]

I've merged this to master and 2.4. I'll target other series and fix-committed master/2.4.

[Lebbeous Fogle-Weekley (lebbeous)]

This tests successfully on a rel_2_3 build also, insofar as one, or at most two, of the "spammed" search queries reach the database and add to load there.

Merging to rel_2_3 and rel_2_2, which will make this whole thing "Fix Committed"

[Lebbeous Fogle-Weekley (lebbeous)]

Also, since we didn't decide to go with a sec announcement or a secret merge-while-releasing process this time, should we mark this bug not-security?