| 2010-07-25 12:18:50 |
Robert Collins |
description |
[Problem]
Bug #317094 has been consistently generating a HTTP Error 503: Service Unavailable over the last week when running a script that processes its attachments.
[Discussion]
Cronned runs of process-release-tagging.py on May 14, 15, 16, 17, 18, 19, and 20 all failed midway through with 503 Service Unavailable errors. Of those runs, all but two failed while processing bug 317094. The other two failed prior to reaching 317094, perhaps for some other reason.
This bug report has over 200 comments, and also has an unusually large number of attachments from a range of different people. I notice if I try to view all the comments from the web interface it times out after a few moments. (I tested both edge and production.)
I think it's fairly well known that bugs with >400 comments tend to time out, but ones with only 216 comments generally are okay. So I'm more wondering if it has something to do with the large number of attachments on this bug report.
When accessing the comments from the web interface, the Oops looks like this
Timeout error
Sorry, something just went wrong in Launchpad.
We’ve recorded what happened, and we’ll fix it as soon as possible. Apologies for the inconvenience.
Trying again in a couple of minutes might work.
(Error ID: OOPS-1601G2848)
Traceback (most recent call last):
* Module zope.publisher.publish, line 134, in publish
* Module canonical.launchpad.webapp.publication, line 428, in callObject
return mapply(ob, request.getPositionalArguments(), request)
* Module zope.publisher.publish, line 109, in mapply
__traceback_info__: <security proxied zope.browserpage.metaconfigure.SimpleViewClass from /srv/launchpad.net/production/launchpad-rev-9342/lib/lp/bugs/browser/../templates/bugtask-index.pt instance at 0x21a0f590>
* Module zope.publisher.publish, line 115, in debug_call
* Module canonical.launchpad.webapp.publisher, line 279, in __call__
return self.render()
* Module lp.bugs.browser.bugtask, line 628, in render
return LaunchpadView.render(self)
* Module canonical.launchpad.webapp.publisher, line 264, in render
return self.template()
* Module zope.app.pagetemplate.viewpagetemplatefile, line 83, in __call__
* Module zope.app.pagetemplate.viewpagetemplatefile, line 51, in __call__
* Module zope.pagetemplate.pagetemplate, line 115, in pt_render
* Module zope.tal.talinterpreter, line 271, in __call__
* Module zope.tal.talinterpreter, line 343, in interpret
* Module zope.tal.talinterpreter, line 888, in do_useMacro
* Module zope.tal.talinterpreter, line 343, in interpret
* Module zope.tal.talinterpreter, line 533, in do_optTag_tal
* Module zope.tal.talinterpreter, line 518, in do_optTag
* Module zope.tal.talinterpreter, line 513, in no_tag
...
* Module canonical.launchpad.webapp.adapter, line 464, in connection_raw_execute
connection, raw_cursor, statement, params)
* Module storm.tracer, line 49, in connection_raw_execute
TimeoutError: 'SELECT LibraryFileContent.datecreated, LibraryFileContent.filesize, LibraryFileContent.id, LibraryFileContent.md5, LibraryFileContent.sha1 FROM LibraryFileContent WHERE LibraryFileContent.id = %s LIMIT 1', [<storm.variables.IntVariable object at 0x2aaab7487578>]<br /> |
[Problem]
Bug #317094 has been consistently generating a HTTP Error 503: Service Unavailable over the last week when running a script that processes its attachments.
[Discussion]
Cronned runs of process-release-tagging.py on May 14, 15, 16, 17, 18, 19, and 20 all failed midway through with 503 Service Unavailable errors. Of those runs, all but two failed while processing bug 317094. The other two failed prior to reaching 317094, perhaps for some other reason.
This bug report has over 200 comments, and also has an unusually large number of attachments from a range of different people. I notice if I try to view all the comments from the web interface it times out after a few moments. (I tested both edge and production.)
I think it's fairly well known that bugs with >400 comments tend to time out, but ones with only 216 comments generally are okay. So I'm more wondering if it has something to do with the large number of attachments on this bug report.
When accessing the comments from the web interface, the Oops looks like this
Timeout error
Sorry, something just went wrong in Launchpad.
We’ve recorded what happened, and we’ll fix it as soon as possible. Apologies for the inconvenience.
Trying again in a couple of minutes might work.
(Error ID: OOPS-1601G2848)
Traceback (most recent call last):
* Module zope.publisher.publish, line 134, in publish
...
* Module canonical.launchpad.webapp.adapter, line 464, in connection_raw_execute
connection, raw_cursor, statement, params)
* Module storm.tracer, line 49, in connection_raw_execute
TimeoutError: 'SELECT LibraryFileContent.datecreated, LibraryFileContent.filesize, LibraryFileContent.id, LibraryFileContent.md5, LibraryFileContent.sha1 FROM LibraryFileContent WHERE LibraryFileContent.id = %s LIMIT 1', [<storm.variables.IntVariable object at 0x2aaab7487578>]<br />
SQL time: 5202 ms
Non-sql time: 15510 ms
Total time: 20712 ms
Statement Count: 506
no long sql statements here - there is however a big jump:
348. 8717 44ms launchpad-main-slave SELECT BugAttachment.bug, BugAttachment.libraryfile, BugAttachment.id, BugAttachment.libraryfile, BugAttachment.message, BugAttachment.title, BugAttachment.type FROM BugAttachment, LibraryFileAlias WHERE BugAttachment.bug = %s AND BugAttachment.libraryfile = LibraryFileAlias.id AND NOT (LibraryFileAlias.content IS NULL) ORDER BY BugAttachment.id
349. 15868 62ms launchpad-main-slave SELECT BugTag.bug, BugTag.id, BugTag.tag FROM BugTag WHERE BugTag.bug = %s ORDER BY BugTag.tag
That is, between getting the attachment data, and starting onto tags, there is a 5 second 'nonsql' gap - could be talking to librarian or something I guess. |
|
