| 2010-11-12 00:34:59 |
Michael Hudson-Doyle |
bug |
|
|
added bug |
| 2010-11-12 01:04:55 |
Tim Penhey |
tags |
|
codehosting-ssh xmlrpc |
|
| 2010-11-12 01:05:00 |
Tim Penhey |
launchpad-code: status |
New |
Triaged |
|
| 2010-11-12 01:05:02 |
Tim Penhey |
launchpad-code: importance |
Undecided |
High |
|
| 2010-11-12 06:00:33 |
Bilal Akhtar |
bug |
|
|
added subscriber Bilal Akhtar |
| 2010-11-12 12:32:21 |
Jonathan Lange |
launchpad-code: importance |
High |
Critical |
|
| 2010-11-12 18:57:28 |
Francis J. Lacoste |
launchpad-code: importance |
Critical |
High |
|
| 2010-11-12 22:55:02 |
Robert Collins |
summary |
bzr push occasionally reports AssertionError on terminal |
bzr push reporting AssertionError on terminal |
|
| 2010-11-12 22:55:34 |
Robert Collins |
description |
This just happened to me (twice):
mwh@grond:testtools$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~mwhudson/launchpad/testtools
Traceback (most recent call last):
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
self.result = callback(self.result, *args, **kw)
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/lib/lp/codehosting/vfs/branchfs.py", line 688, in got_path_info
control_string, branch_string, repository_string)
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/lib/lp/codehosting/vfs/branchfsclient.py", line 122, in branchChanged
repository_string)
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/lib/lp/services/twistedsupport/xmlrpc.py", line 50, in callRemote
method_name, *args, **kwargs)
--- <exception caught here> ---
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 125, in maybeDeferred
result = f(*args, **kw)
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/lib/lp/services/twistedsupport/xmlrpc.py", line 35, in callRemote
return getattr(self._proxy, method_name)(*args, **kwargs)
File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
verbose=self.__verbose
File "/usr/lib/python2.6/xmlrpclib.py", line 1253, in request
return self._parse_response(h.getfile(), sock)
File "/usr/lib/python2.6/xmlrpclib.py", line 1392, in _parse_response
return u.close()
File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'Unexpected Zope exception: AssertionError: '>
No new revisions to push.
HPSS calls: 12 (0 vfs) SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)
OOPS-1777XMLP119 is if not my oops, an example of the same problem.
Things to note:
1) the traceback is being printed to stderr by the bzr serve process on the server
2) the assertionerror is being raised by a call to transaction.doom() that's in an "except (RequestExpired, TimeoutError):" block
3) the sql log in the oops reports a gap of 15s between sql requests (so a timeout is legitimate, although something funny is going on)
4) that assertionerror is raised when the transaction is not "active" or already "doomed". The other possible statuses are "committing", "committed" or "commitfailed". I don't know which. |
mwh@grond:testtools$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~mwhudson/launchpad/testtools
Traceback (most recent call last):
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
...
File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'Unexpected Zope exception: AssertionError: '>
No new revisions to push.
HPSS calls: 12 (0 vfs) SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)
OOPS-1777XMLP119 is if not my oops, an example of the same problem.
Things to note:
1) the traceback is being printed to stderr by the bzr serve process on the server
2) the assertionerror is being raised by a call to transaction.doom() that's in an "except (RequestExpired, TimeoutError):" block
3) the sql log in the oops reports a gap of 15s between sql requests (so a timeout is legitimate, although something funny is going on)
4) that assertionerror is raised when the transaction is not "active" or already "doomed". The other possible statuses are "committing", "committed" or "commitfailed". I don't know which.
The XMLRPC internal server is suffering a high rate of OOPSes. |
|
| 2010-11-12 22:58:56 |
Robert Collins |
description |
mwh@grond:testtools$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~mwhudson/launchpad/testtools
Traceback (most recent call last):
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
...
File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'Unexpected Zope exception: AssertionError: '>
No new revisions to push.
HPSS calls: 12 (0 vfs) SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)
OOPS-1777XMLP119 is if not my oops, an example of the same problem.
Things to note:
1) the traceback is being printed to stderr by the bzr serve process on the server
2) the assertionerror is being raised by a call to transaction.doom() that's in an "except (RequestExpired, TimeoutError):" block
3) the sql log in the oops reports a gap of 15s between sql requests (so a timeout is legitimate, although something funny is going on)
4) that assertionerror is raised when the transaction is not "active" or already "doomed". The other possible statuses are "committing", "committed" or "commitfailed". I don't know which.
The XMLRPC internal server is suffering a high rate of OOPSes. |
mwh@grond:testtools$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~mwhudson/launchpad/testtools
Traceback (most recent call last):
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
...
File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'Unexpected Zope exception: AssertionError: '>
No new revisions to push.
HPSS calls: 12 (0 vfs) SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)
OOPS-1777XMLP119 is if not my oops, an example of the same problem.
Things to note:
1) the traceback is being printed to stderr by the bzr serve process on the server
2) the assertionerror is being raised by a call to transaction.doom() that's in an "except (RequestExpired, TimeoutError):" block
3) the sql log in the oops reports a gap of 15s between sql requests (so a timeout is legitimate, although something funny is going on)
4) that assertionerror is raised when the transaction is not "active" or already "doomed". The other possible statuses are "committing", "committed" or "commitfailed". I don't know which.
The XMLRPC internal server is suffering a high rate of OOPSes / timeouts and its these that are being shown on the codehosting front end and passed onto the user.
We believe it is a load/timeout issue on one specific internal server.
|
|
| 2010-11-13 03:51:13 |
Robert Collins |
description |
mwh@grond:testtools$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~mwhudson/launchpad/testtools
Traceback (most recent call last):
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
...
File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'Unexpected Zope exception: AssertionError: '>
No new revisions to push.
HPSS calls: 12 (0 vfs) SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)
OOPS-1777XMLP119 is if not my oops, an example of the same problem.
Things to note:
1) the traceback is being printed to stderr by the bzr serve process on the server
2) the assertionerror is being raised by a call to transaction.doom() that's in an "except (RequestExpired, TimeoutError):" block
3) the sql log in the oops reports a gap of 15s between sql requests (so a timeout is legitimate, although something funny is going on)
4) that assertionerror is raised when the transaction is not "active" or already "doomed". The other possible statuses are "committing", "committed" or "commitfailed". I don't know which.
The XMLRPC internal server is suffering a high rate of OOPSes / timeouts and its these that are being shown on the codehosting front end and passed onto the user.
We believe it is a load/timeout issue on one specific internal server.
|
mwh@grond:testtools$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~mwhudson/launchpad/testtools
Traceback (most recent call last):
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
...
File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'Unexpected Zope exception: AssertionError: '>
No new revisions to push.
HPSS calls: 12 (0 vfs) SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)
OOPS-1777XMLP119 is if not my oops, an example of the same problem.
Things to note:
1) the traceback is being printed to stderr by the bzr serve process on the server
2) the assertionerror is being raised by a call to transaction.doom() that's in an "except (RequestExpired, TimeoutError):" block
3) the sql log in the oops reports a gap of 15s between sql requests (so a timeout is legitimate, although something funny is going on)
4) that assertionerror is raised when the transaction is not "active" or already "doomed". The other possible statuses are "committing", "committed" or "commitfailed". I don't know which.
The XMLRPC internal server is suffering a high rate of OOPSes / timeouts and its these that are being shown on the codehosting front end and passed onto the user.
We believe it is a load/timeout issue on one specific internal server.
https://rt.admin.canonical.com/Ticket/Display.html?id=41465 is a planned deployment change to get more resources for these queries.
|
|
| 2010-11-13 05:46:51 |
Cody A.W. Somerville |
tags |
codehosting-ssh xmlrpc |
codehosting-ssh oem-services xmlrpc |
|
| 2010-11-13 09:53:41 |
Robert Collins |
launchpad-code: status |
Triaged |
In Progress |
|
| 2010-11-13 09:56:56 |
Robert Collins |
description |
mwh@grond:testtools$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~mwhudson/launchpad/testtools
Traceback (most recent call last):
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
...
File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'Unexpected Zope exception: AssertionError: '>
No new revisions to push.
HPSS calls: 12 (0 vfs) SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)
OOPS-1777XMLP119 is if not my oops, an example of the same problem.
Things to note:
1) the traceback is being printed to stderr by the bzr serve process on the server
2) the assertionerror is being raised by a call to transaction.doom() that's in an "except (RequestExpired, TimeoutError):" block
3) the sql log in the oops reports a gap of 15s between sql requests (so a timeout is legitimate, although something funny is going on)
4) that assertionerror is raised when the transaction is not "active" or already "doomed". The other possible statuses are "committing", "committed" or "commitfailed". I don't know which.
The XMLRPC internal server is suffering a high rate of OOPSes / timeouts and its these that are being shown on the codehosting front end and passed onto the user.
We believe it is a load/timeout issue on one specific internal server.
https://rt.admin.canonical.com/Ticket/Display.html?id=41465 is a planned deployment change to get more resources for these queries.
|
mwh@grond:testtools$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~mwhudson/launchpad/testtools
Traceback (most recent call last):
File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
...
File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'Unexpected Zope exception: AssertionError: '>
No new revisions to push.
HPSS calls: 12 (0 vfs) SmartSSHClientMedium(bzr+ssh://mwhudson@bazaar.launchpad.net/)
OOPS-1777XMLP119 is if not my oops, an example of the same problem.
Things to note:
1) the traceback is being printed to stderr by the bzr serve process on the server
2) the assertionerror is being raised by a call to transaction.doom() that's in an "except (RequestExpired, TimeoutError):" block
3) the sql log in the oops reports a gap of 15s between sql requests (so a timeout is legitimate, although something funny is going on)
4) that assertionerror is raised when the transaction is not "active" or already "doomed". The other possible statuses are "committing", "committed" or "commitfailed". I don't know which.
The XMLRPC internal server was suffering a high rate of OOPSes / timeouts and its these that are being shown on the codehosting front end and passed onto the user.
We believe it is a load/timeout issue on one specific internal server.
=======
We think this is fixed: we've reconfigured the backend xmlrpc service to be served from our primary appserver cluster, so there are now 13 times the resources available for it; we've further fine tuning and capacity management to do, but the basic situation should be remedied.
*please* comment here if you encounter zope assertion error, so that we can tell the bug isn't fixed. |
|
| 2010-11-13 13:48:54 |
Colin Watson |
bug |
|
|
added subscriber Colin Watson |
| 2010-11-16 14:59:21 |
Guewen Baconnier @ Camptocamp |
bug |
|
|
added subscriber Guewen Baconnier @ Camptocamp |
| 2010-11-18 00:49:55 |
Robert Collins |
launchpad-code: status |
In Progress |
Fix Released |
|
