codebrowse responding to debug requests; 500 erroring on everything else

[Steve McInerney]

We've hit this a couple of times now:
* nagios alerts with a CRITICAL on codebrowse
"HTTP CRITICAL - Invalid HTTP response received from host"
* the -debug check responds fine:
               url time running time since last activity
 http://localhost:8080/thread-debug 7.15255737305e-06 7.15255737305e-06
* manual browsing of other code trees gives similar errors to nagios
* only a restart seems to cure the woe

We have a core dump on guava from yesterdays fail: guava:~/core.32380.gz
300Mb file compressed.

[Steve McInerney (spm)]

Setting to high: we've had at least 4 of these incidents in the past 10ish days.

[Max Kanat-Alexander (mkanat)]

Okay. I need the logs starting from 10 minutes before the hang through when the service is restarted. It would also be helpful to know the exact timestamp at which nagios started reporting the problem so that I can cross-index that to the logs.

[Max Kanat-Alexander (mkanat)]

I suspect that this was the result of a race condition that we actually already fixed, but that launchpad had incorrectly not merged.