lp.codehosting.tests.test_bzrutils.TestGetBranchStackedOnURL.testGetBranchStackedOnUrlNotStacked(RemoteBranchFormat-v2) fails rarely/intermittently in parallel tests

Bug #998040 reported by Gary Poster on 2012-05-11
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Unassigned

Bug Description

======================================================================
ERROR: lp.codehosting.tests.test_bzrutils.TestGetBranchStackedOnURL.testGetBranchStackedOnUrlNotStacked(RemoteBranchFormat-v2)
worker ID: worker-3
----------------------------------------------------------------------
Traceback (most recent call last):
_StringException: Empty attachments:
  log

Traceback (most recent call last):
  File "/var/lib/buildbot/slaves/slave/lucid-devel/build/lib/lp/codehosting/tests/test_bzrutils.py", line 95, in testGetBranchStackedOnUrlNotStacked
    branch = self.make_branch('source')
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/tests/__init__.py", line 2632, in make_branch
    repo = self.make_repository(relpath, format=format)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/tests/__init__.py", line 2689, in make_repository
    made_control = self.make_bzrdir(relpath, format=format)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/tests/per_controldir/__init__.py", line 74, in make_bzrdir
    relpath, format=format)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/tests/__init__.py", line 2674, in make_bzrdir
    t.ensure_base()
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/transport/__init__.py", line 385, in ensure_base
    self.mkdir('.')
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/transport/remote.py", line 242, in mkdir
    self._serialise_optional_mode(mode))
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/transport/remote.py", line 182, in _call2
    return self._client.call(method, *args)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/client.py", line 132, in call
    result, protocol = self.call_expecting_body(method, *args)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/client.py", line 145, in call_expecting_body
    method, args, expect_response_body=True)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/client.py", line 79, in _call_and_read_response
    readv_body=readv_body, body_stream=body_stream)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/client.py", line 63, in _send_request
    encoder.call(method, *args)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/protocol.py", line 630, in call
    self._write_args(args)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/protocol.py", line 801, in _write_args
    self._write_protocol_version()
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/protocol.py", line 851, in _write_protocol_version
    self._request.accept_bytes(self.request_marker)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/medium.py", line 396, in accept_bytes
    self._accept_bytes(bytes)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/medium.py", line 1002, in _accept_bytes
    self._medium._accept_bytes(bytes)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/medium.py", line 886, in _accept_bytes
    osutils.send_all(self._socket, bytes, self._report_activity)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/osutils.py", line 2091, in send_all
    report_activity(sent, 'write')
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/smart/medium.py", line 176, in _report_activity
    ui.ui_factory.report_transport_activity(self, bytes, direction)
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/lazy_import.py", line 121, in __getattribute__
    obj = _replace()
  File "/var/lib/buildbot/launchpad/dependencies/eggs/bzr-2.5.0dev2_r6152-py2.6-linux-i686.egg/bzrlib/lazy_import.py", line 89, in _replace
    extra=e)
IllegalUseOfScopeReplacer: ScopeReplacer object 'ui' was used incorrectly: Object already cleaned up, did you assign it to another variable?: _scope

Worker 3 was then hosed for 196 other tests, every single one with exceptions like this one:

======================================================================
ERROR: lp.scripts.tests.test_sphinxdocs.TestSphinxDocumentation.test_docs_build_without_error
worker ID: worker-3
----------------------------------------------------------------------
garbage: {{{[<bzrlib.smart.medium.SmartTCPClientMedium object at 0x13685c6c>]}}}

Worker 3 test list: http://pastebin.ubuntu.com/981669/

Related branches

Brad Crittenden (bac) on 2012-05-11
Changed in launchpad:
assignee: nobody → Brad Crittenden (bac)
status: Triaged → In Progress
Brad Crittenden (bac) wrote :

I've tried repeatedly to reproduce this error but cannot. I've run the ordered tests within the layer where the failure occurred over 1000 times with no failures. I've run the entire set of tests shown in the worker 3 test list (up to the failed test) with no errors.

Changed in launchpad:
status: In Progress → Incomplete
assignee: Brad Crittenden (bac) → nobody
status: Incomplete → Triaged
Brad Crittenden (bac) wrote :

Robert pointed to bug 593536 as a possible cause. WIll follow up with bzr guys.

Brad Crittenden (bac) wrote :

<lifeless> bac: revno: 6426 [merge]
committer: Patch Queue Manager <email address hidden>
branch nick: +trunk
timestamp: Thu 2012-01-05 11:06:47 +0000
message:
 (gz) Make lazy imports proxy by default so that concurrent resolution is
  robust (Martin Packman)
bac: so it looks like its fixed as of rev 6426
bac: so 2.6 has the fix
bac: we could upgrade to 2.6bwhatever, or backport the fix.

Martin Packman (gz) wrote :

Just bzr 2.5.0 should be enough, that includes the relevant fix. Currently launchpad uses bzr-2.5.0dev2_r6152 and Jelmer tried bumping the version shortly after the Bazaar release. Unfortunately he ran into some issues where launchpad was using deprecated methods, and trying to fix those broke various other things so the effort stalled. Either going back and resolving those problems, or he suggested cherry picking just this fix for launchpad, would be a way forwards.

Gary Poster (gary) on 2012-05-23
Changed in launchpad:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers