TooManyConcurrentRequests error when ssh connection fails (bzr crashes when pulling)

Bug #246233 reported by Stavros Korokithakis
48
This bug affects 3 people
Affects Status Importance Assigned to Milestone
5-a-day-data
Undecided
Unassigned
Bazaar
High
Andrew Bennetts

Bug Description

When a "bzr merge" or "bzr pull" uses a bzr+ssh:// or bzr:// URL, and connecting to that URL fails (e.g. due to SSH auth failure), bzr gives a TooManyConcurrentRequests traceback. Here's an example from the original bug report:

"""
skorokithakis@belladonna:~/Desktop/omnisync$ bzr pull lp:omnisync
Permission denied (publickey).
bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x88ff10c>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 646, in run
    possible_transports=possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 107, in open
    possible_transports=possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 667, in open
    return BzrDir.open_from_transport(t, _unsupported=_unsupported)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 704, in open_from_transport
    redirected)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 1616, in do_catching_redirections
    return action(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 681, in find_format
    transport, _server_formats=_server_formats)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 1440, in find_format
    return format.probe_transport(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 2398, in probe_transport
    server_version = medium.protocol_version()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 417, in protocol_version
    medium_request = self.get_request()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 476, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 651, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x88ff10c>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

bzr 1.5 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'pull', 'lp:omnisync']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.5.0]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

I was trying to pull a repo while logged into launchpad but without an SSH key, and it crashed the second time.
"""

Bzr should instead be raising the original ConnectionReset error.

Related branches

Revision history for this message
Hew (hew) wrote :

I'm getting this as well when using 5-a-day on Ubuntu Intrepid. It's reproducible.

bzr/intrepid uptodate 1.5-1

Changed in bzr:
status: New → Confirmed
Revision history for this message
Jim Fisher (jedijf) wrote :

Trying to register for GlobalBugJam same thing.

Applet fails too.

jimf@jimf-ibook:/usr/bin$ sudo 5-a-day --update
[sudo] password for jimf:
Permission denied (publickey).
bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x10468870>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 633, in run
    branch_from = Branch.open_from_transport(location_transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 115, in open_from_transport
    control = bzrdir.BzrDir.open_from_transport(transport, _unsupported)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 688, in open_from_transport
    redirected)
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/__init__.py", line 1664, in do_catching_redirections
    return action(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 665, in find_format
    transport, _server_formats=_server_formats)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 1422, in find_format
    return format.probe_transport(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 2406, in probe_transport
    request = client.get_request()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 417, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 592, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x10468870>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

bzr 1.3.1 on python 2.5.2.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'pull', 'bzr+ssh://<email address hidden>/~jedijf/5-a-day-data/main']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.
bzr failed with error code 1024

Jim Fisher (jedijf)
Changed in 5-a-day-data:
status: New → Invalid
Martin Pool (mbp)
Changed in bzr:
importance: Undecided → High
Revision history for this message
Frits Jalvingh (fjalvingh) wrote :
Download full text (3.8 KiB)

Same here, for a 'merge' using a DNS name that could not be resolved, using the smart server:
$ bzr merge bzr://bzr/vp-3.1-hot-installer
The dns name 'bzr' was not known on my machine and it caused the same problem. Perhaps any connection error causes the above message?

bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartTCPClientMedium object at 0x8473d0c>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 857, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 2901, in run
    location, revision, remember, possible_transports, pb)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 2979, in _get_merger_from_branch
    possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 142, in open_containing
    possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 826, in open_containing
    return BzrDir.open_containing_from_transport(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 847, in open_containing_from_transport
    result = BzrDir.open_from_transport(a_transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 806, in open_from_transport
    return format.open(transport, _found=True)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 1733, in open
    return self._open(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 2613, in _open
    return remote.RemoteBzrDir(transport)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 76, in __init__
    response = self._client.call('BzrDir.open', path)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 111, in call
    result, protocol = self.call_expecting_body(method, *args)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 124, in call_expecting_body
    method, args, expect_response_body=True)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 69, in _call_and_read_response
    readv_body=readv_body)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 39, in _send_request
    protocol_version)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/client.py", line 94, in _construct_protocol
    request = self._medium.get_request()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 589, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 764, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartTCPClientMedium object at 0x8473d0c>' has reached its concurrent request...

Read more...

Revision history for this message
James Westby (james-w) wrote :

Hi,

These errors are usually secondary, meaning that there was first
the real problem, and as bzr tried to clean up it triggered this problem.

Another user reported this problem with 5-a-day, and said that it was
due to either not having an SSH key registered for their launchpad account,
or having the wrong one, could you check whether that is the case for you?

Also, could you check that your username in ~/.5-a-day matches your
launchpad username?

Thanks,

James

Revision history for this message
Andrew Bennetts (spiv) wrote :

I can reproduce this.

The problem is that the first thing bzr tries to do with a pull or merge is try to fetch the URL as file in case it is a bundle. If this fails due to being unable to connect (e.g. because SSH auth fails), the ConnectionReset error is intercepted by bzrlib.bundle.read_mergeable_from_transport's "except TransportError" and turned into a NotABundle error. Bzr then continues to try open the URL as a BzrDir, rather than immediately giving up and reporting the connection problem to the user.

There's two problems here, I think. First is that ConnectionReset isn't like the other errors that code wants to catch (e.g. the URL is a directory thus the 'get' fails); the connection is fundamentally broken and unusable, so continuing with the BzrDir opening is the wrong things to do here.

Secondly, if a stream-based smart medium has failed due to ConnectionReset, it should probably keep raising ConnectionReset if some code tries to keep using it, rather than getting into the state that produces the TooManyConcurrentRequests error. Ideally once a medium raises ConnectionReset it should not be used again, but if something does then reraisng ConnectionReset is more useful than TooManyConcurrentRequests.

Fixing the first part of this should be easy, so I'll do that for 1.8.

Changed in bzr:
assignee: nobody → spiv
milestone: none → 1.8
description: updated
Revision history for this message
Andrew Bennetts (spiv) wrote :

I've sent a patch to fix this to the mailing list for review.

Changed in bzr:
status: Confirmed → Fix Committed
Andrew Bennetts (spiv)
Changed in bzr:
status: Fix Committed → Fix Released
Revision history for this message
Doug Lee (dgl-dlee) wrote :

Not sure of the current status of this one, but it also applies to commits to a bound branch when the parent bzr+ssh branch is inaccessible. I'm using Bzr 1.10 on Cygwin, so a bit old; apologies if my info is out of date. I just got this when committing to a bound branch when the parent's server happened to be temporarily offline (names/project references genericized):

Doug 2% bzr commit -m "Skeleton file set."
ssh: connect to host bzr-central.office.com port 22: Connection timed out
bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x7fa1ceac>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "commands.py", line 893, in run_bzr_catch_errors
  File "commands.py", line 839, in run_bzr
  File "commands.py", line 539, in run_argv_aliases
  File "builtins.py", line 2435, in run
  File "decorators.py", line 192, in write_locked
  File "workingtree_4.py", line 235, in commit
  File "decorators.py", line 192, in write_locked
  File "mutabletree.py", line 202, in commit
  File "commit.py", line 303, in commit
  File "commit.py", line 464, in _check_bound_branch
  File "decorators.py", line 138, in read_locked
  File "branch.py", line 1984, in get_master_branch
  File "branch.py", line 123, in open
  File "bzrdir.py", line 766, in open
  File "bzrdir.py", line 808, in open_from_transport
  File "bzrdir.py", line 1770, in open
  File "bzrdir.py", line 2646, in _open
  File "remote.py", line 93, in __init__
  File "remote.py", line 51, in _call
  File "client.py", line 122, in call
  File "client.py", line 135, in call_expecting_body
  File "client.py", line 80, in _call_and_read_response
  File "client.py", line 42, in _send_request
  File "client.py", line 105, in _construct_protocol
  File "medium.py", line 672, in get_request
  File "medium.py", line 870, in __init__
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x7fa1ceac>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

bzr 1.10 on python 2.5.2 (cygwin)
arguments: ['/usr/bin/bzr', 'commit', '-m', 'Skeleton file set.']
encoding: 'US-ASCII', fsenc: 'US-ASCII', lang: None
plugins:
  bookmarks /home/Doug/.bazaar/plugins/bookmarks [unknown]
  bzrtools /home/Doug/.bazaar/plugins/bzrtools [1.10]
  cvsps_import /home/Doug/.bazaar/plugins/cvsps_import [unknown]
  email /home/Doug/.bazaar/plugins/email [unknown]
  fastimport /home/Doug/.bazaar/plugins/fastimport [unknown]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  service /home/Doug/.bazaar/plugins/service [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.
Doug 2%

Martin Pool (mbp)
Changed in bzr:
status: Fix Released → Confirmed
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers