importer doesn't cope with broken ssh pipe

Bug #828664 reported by James Page on 2011-08-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Distributed Development
High
Unassigned

Bug Description

http://package-import.ubuntu.com/status/jenkins.html#2011-08-01 18:15:35.769737

Please could this package be imported as I need to make a couple of fixes.

Cheers

James

Failed at 2011-08-01 18:15:35.769737

Connection to bazaar.launchpad.net closed by remote host.
Traceback (most recent call last):
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1124, in <module>
    only_before=options.only_before))
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1058, in main
    push_branches_back(package, temp_dir, bstore, possible_transports, local_branches)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 712, in push_branches_back
    overwrite=overwrite)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 388, in push_branch
    dir_to = bzrdir.BzrDir.open_from_transport(to_transport)
  File "/usr/lib/pymodules/python2.6/bzrlib/bzrdir.py", line 750, in open_from_transport
    return format.open(transport, _found=True)
  File "/usr/lib/pymodules/python2.6/bzrlib/bzrdir.py", line 1697, in open
    return self._open(transport)
  File "/usr/lib/pymodules/python2.6/bzrlib/bzrdir.py", line 2884, in _open
    return remote.RemoteBzrDir(transport, self)
  File "/usr/lib/pymodules/python2.6/bzrlib/remote.py", line 120, in __init__
    self._probe_bzrdir()
  File "/usr/lib/pymodules/python2.6/bzrlib/remote.py", line 132, in _probe_bzrdir
    self._rpc_open_2_1(path)
  File "/usr/lib/pymodules/python2.6/bzrlib/remote.py", line 139, in _rpc_open_2_1
    response = self._call('BzrDir.open_2.1', path)
  File "/usr/lib/pymodules/python2.6/bzrlib/remote.py", line 57, in _call
    return self._client.call(method, *args)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 132, in call
    result, protocol = self.call_expecting_body(method, *args)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 145, in call_expecting_body
    method, args, expect_response_body=True)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 79, in _call_and_read_response
    readv_body=readv_body, body_stream=body_stream)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 45, in _send_request
    protocol_version)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 115, in _construct_protocol
    request = self._medium.get_request()
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/medium.py", line 710, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/medium.py", line 968, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
bzrlib.errors.TooManyConcurrentRequests: The medium 'SmartSSHClientMedium(bzr+ssh://<email address hidden>/)' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Martin Pool (mbp) on 2011-08-18
Changed in udd:
status: New → Confirmed
importance: Undecided → High
description: updated
Martin Pool (mbp) on 2011-08-18
Changed in udd:
assignee: nobody → Martin Pool (mbp)
status: Confirmed → In Progress
Martin Pool (mbp) wrote :

@James ubuntu:jenkins is now up to date (checked by branching it which implicitly checks against publication) - thanks for the report and sorry for the disruption.

I'm attaching the log which shows the failure is

136.161 Committed revid <email address hidden> as revno 2.
137.614 Marking version 1.409.1-0ubuntu1 in oneiric at revid <email address hidden>
141.324 refresh_possible_transports dropping stale transport <bzrlib.transport.remote.RemoteSSHTransport url=bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/jenkins/oneiric/> ([Errno 32] Broken pipe)
141.324 refresh_possible_transports encountered a smart transport with outstanding request: <bzrlib.transport.remote.RemoteSSHTransport url=bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/debian/woody/jenkins/woody/>
... and on and on.

So I think the bug is: launchpad dropped the connection (or maybe the network glitched or something else.) What the udd importer should have done is probably stop at that point and treat it as a transient failure. What did happen is that it kept trying to use the dead ssh connection and tied itself in knots - there is possibly also a bzr bug there that it didn't try to reconnect. (cf bug 819604 and friends).

So, though the immediate thing is fixed by retrying I'll leave this open for that bug.

summary: - Jenkins package failed to import - TooManyConcurrentRequests
+ importer doesn't cope with broken ssh pipe
Martin Pool (mbp) wrote :

eglibc php5 and ubuntu-docs all had similar errors so I've requeued them too

Martin Pool (mbp) wrote :

ubuntu-docs seems to be ok, but eglibc and php5 failed with a similar error.

so for eglibc it also seems to have run into a stale transport and not to be handling it properly:

26225.211 refresh_possible_transports dropping stale transport <bzrlib.transport.remote.RemoteSSHTransport url=bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/eglibc/oneiric/> ([Errno 32] Broken pipe)
26225.211 refresh_possible_transports encountered a smart transport with outstanding request: <bzrlib.transport.remote.RemoteSSHTransport url=bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/debian/sid/eglibc/sid/>

Martin Pool (mbp) wrote :

I didn't really start on this, and it isn't affecting many packages at the moment, so I'm throwing it back. It is still blocking eglibc and php5 so perhaps it's related to a timeout on large packages?

Changed in udd:
assignee: Martin Pool (mbp) → nobody
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers