coverstore has intermitent 500 errors

[samuel-archive]

Book cover fetches are randomly getting 500 errors.
Retries of the GET almost always are successful.

In a 100 second sample this afternoon I saw 11 500 errors from coverstore.

example from that sample:

ia331532$ tail -s 0.1 --follow=name /var/log/lighttpd/current-access.log | egrep ' 50[0-9] '

yeilds lines like this:
0.36.56.35 covers.openlibrary.org - [10/May/2010:23:20:52 +0000] "GET /b/olid/OL130247M-M.jpg HTTP/1.0" 500 21 "http://openlibrary.org/authors/OL85833A/Salvador_Novo" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6.4; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2)" 2

The error log for this 500 is:
2010-05-10 23:20:52: (mod_fastcgi.c.2610) FastCGI-stderr: Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/web.py-0.33-py2.5.egg/web/application.py", line 242, in process
    return self.handle()
  File "/usr/lib/python2.5/site-packages/web.py-0.33-py2.5.egg/web/application.py", line 233, in handle
    return self._delegate(fn, self.fvars, args)
  File "/usr/lib/python2.5/site-packages/web.py-0.33-py2.5.egg/web/application.py", line 412, in _delegate
    return handle_class(cls)
  File "/usr/lib/python2.5/site-packages/web.py-0.33-py2.5.egg/web/application.py", line 387, in handle_class
    return tocall(*args)
  File "/opt/openlibrary/production/openlibrary/coverstore/code.py", line 141, in GET
    d = _query(category, key, value)
  File "/opt/openlibrary/production/openlibrary/coverstore/code.py", line 32, in _query
    result = db.query(category, value, limit=1)
  File "/opt/openlibrary/production/openlibrary/coverstore/db.py", line 64, in query
    return result.list()
  File "/usr/lib/python2.5/site-packages/web.py-0.33-py2.5.egg/web/db.py", line 615, in <lambda>
    for x in db_cursor.fetchall()]
InterfaceError: cursor already closed

[George (george-archive)]

This happened twice today, fwiw. Andy started once from NAGIOS UI, and I did the second time. No idea why it happened though.

[Anand Chitipothu (anandology)]

On 14-Sep-2010, at 5:48 AM, George wrote:

> This happened twice today, fwiw. Andy started once from NAGIOS UI, and I
> did the second time. No idea why it happened though.

Experimenting with gunicorn[1], a webserver in python. Looking quite good. Will deploy it in a day.

[1]: http://gunicorn.org/

Anand

[Anand Chitipothu (anandology)]

On 14-Sep-2010, at 6:11 PM, Anand Chitipothu wrote:

>
> On 14-Sep-2010, at 5:48 AM, George wrote:
>
>> This happened twice today, fwiw. Andy started once from NAGIOS UI, and I
>> did the second time. No idea why it happened though.
>
> Experimenting with gunicorn[1], a webserver in python. Looking quite
> good. Will deploy it in a day.
>
> [1]: http://gunicorn.org/

Deployed. I'm hoping to see it stable from now on.

Anand

[Anand Chitipothu (anandology)]

Looks quite stable and even the memory usage is quite low.