importer doesn't cope with broken ssh pipe

[James Page]

http://package-import.ubuntu.com/status/jenkins.html#2011-08-01 18:15:35.769737

Please could this package be imported as I need to make a couple of fixes.

Cheers

James

Failed at 2011-08-01 18:15:35.769737

Connection to bazaar.launchpad.net closed by remote host.
Traceback (most recent call last):
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1124, in <module>
    only_before=options.only_before))
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1058, in main
    push_branches_back(package, temp_dir, bstore, possible_transports, local_branches)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 712, in push_branches_back
    overwrite=overwrite)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 388, in push_branch
    dir_to = bzrdir.BzrDir.open_from_transport(to_transport)
  File "/usr/lib/pymodules/python2.6/bzrlib/bzrdir.py", line 750, in open_from_transport
    return format.open(transport, _found=True)
  File "/usr/lib/pymodules/python2.6/bzrlib/bzrdir.py", line 1697, in open
    return self._open(transport)
  File "/usr/lib/pymodules/python2.6/bzrlib/bzrdir.py", line 2884, in _open
    return remote.RemoteBzrDir(transport, self)
  File "/usr/lib/pymodules/python2.6/bzrlib/remote.py", line 120, in __init__
    self._probe_bzrdir()
  File "/usr/lib/pymodules/python2.6/bzrlib/remote.py", line 132, in _probe_bzrdir
    self._rpc_open_2_1(path)
  File "/usr/lib/pymodules/python2.6/bzrlib/remote.py", line 139, in _rpc_open_2_1
    response = self._call('BzrDir.open_2.1', path)
  File "/usr/lib/pymodules/python2.6/bzrlib/remote.py", line 57, in _call
    return self._client.call(method, *args)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 132, in call
    result, protocol = self.call_expecting_body(method, *args)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 145, in call_expecting_body
    method, args, expect_response_body=True)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 79, in _call_and_read_response
    readv_body=readv_body, body_stream=body_stream)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 45, in _send_request
    protocol_version)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/client.py", line 115, in _construct_protocol
    request = self._medium.get_request()
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/medium.py", line 710, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/pymodules/python2.6/bzrlib/smart/medium.py", line 968, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
bzrlib.errors.TooManyConcurrentRequests: The medium 'SmartSSHClientMedium(bzr+ssh://<email address hidden>/)' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

[Martin Pool (mbp)]

@James ubuntu:jenkins is now up to date (checked by branching it which implicitly checks against publication) - thanks for the report and sorry for the disruption.

I'm attaching the log which shows the failure is

136.161 Committed revid <email address hidden> as revno 2.
137.614 Marking version 1.409.1-0ubuntu1 in oneiric at revid <email address hidden>
141.324 refresh_possible_transports dropping stale transport <bzrlib.transport.remote.RemoteSSHTransport url=bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/jenkins/oneiric/> ([Errno 32] Broken pipe)
141.324 refresh_possible_transports encountered a smart transport with outstanding request: <bzrlib.transport.remote.RemoteSSHTransport url=bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/debian/woody/jenkins/woody/>
... and on and on.

So I think the bug is: launchpad dropped the connection (or maybe the network glitched or something else.) What the udd importer should have done is probably stop at that point and treat it as a transient failure. What did happen is that it kept trying to use the dead ssh connection and tied itself in knots - there is possibly also a bzr bug there that it didn't try to reconnect. (cf bug 819604 and friends).

So, though the immediate thing is fixed by retrying I'll leave this open for that bug.

[Martin Pool (mbp)]

eglibc php5 and ubuntu-docs all had similar errors so I've requeued them too

[Martin Pool (mbp)]

ubuntu-docs seems to be ok, but eglibc and php5 failed with a similar error.

so for eglibc it also seems to have run into a stale transport and not to be handling it properly:

26225.211 refresh_possible_transports dropping stale transport <bzrlib.transport.remote.RemoteSSHTransport url=bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/oneiric/eglibc/oneiric/> ([Errno 32] Broken pipe)
26225.211 refresh_possible_transports encountered a smart transport with outstanding request: <bzrlib.transport.remote.RemoteSSHTransport url=bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/debian/sid/eglibc/sid/>

[Martin Pool (mbp)]

I didn't really start on this, and it isn't affecting many packages at the moment, so I'm throwing it back. It is still blocking eglibc and php5 so perhaps it's related to a timeout on large packages?