TooManyConcurrentRequests during unlock when using smart server over ssh

Bug #125784 reported by Jelmer Vernooij
60
Affects Status Importance Assigned to Milestone
Bazaar
Fix Released
High
Andrew Bennetts

Bug Description

  affects /products/bzr-svn

I keep getting concurrency limit errors from the smart server.

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 718,
in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 679,
in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.4/site-packages/bzrlib/commands.py", line 375,
in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.4/site-packages/bzrlib/builtins.py", line 784,
in run
    revision_id=last_revision_id)
  File "/usr/lib/python2.4/site-packages/bzrlib/remote.py", line 518, in
fetch
    return self._real_repository.fetch(
  File "/usr/lib/python2.4/site-packages/bzrlib/repository.py", line
364, in fetch
    return inter.fetch(revision_id=revision_id, pb=pb)
  File "/usr/lib/python2.4/site-packages/bzrlib/decorators.py", line
167, in write_locked
    self.unlock()
  File "/usr/lib/python2.4/site-packages/bzrlib/inter.py", line 110, in
unlock
    self.target.unlock()
  File "/usr/lib/python2.4/site-packages/bzrlib/repository.py", line
387, in unlock
    self.control_files.unlock()
  File "/usr/lib/python2.4/site-packages/bzrlib/lockable_files.py", line
294, in unlock
    self._lock.unlock()
  File "/usr/lib/python2.4/site-packages/bzrlib/lockdir.py", line 246,
in unlock
    self.confirm()
  File "/usr/lib/python2.4/site-packages/bzrlib/lockdir.py", line 320,
in confirm
    info = self.peek()
  File "/usr/lib/python2.4/site-packages/bzrlib/lockdir.py", line 343,
in peek
    info = self._read_info_file(self._held_info_path)
  File "/usr/lib/python2.4/site-packages/bzrlib/lockdir.py", line 333,
in _read_info_file
    return self._parse_info(self.transport.get(path))
  File "/usr/lib/python2.4/site-packages/bzrlib/transport/remote.py",
line 205, in get
    return StringIO(self.get_bytes(relpath))
  File "/usr/lib/python2.4/site-packages/bzrlib/transport/remote.py",
line 209, in get_bytes
    request = self._medium.get_request()
  File "/usr/lib/python2.4/site-packages/bzrlib/smart/medium.py", line
412, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/python2.4/site-packages/bzrlib/smart/medium.py", line
566, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
TooManyConcurrentRequests: The medium
'<bzrlib.smart.medium.SmartSSHClientMedium object at 0xb7732f0c>' has
reached its concurrent request limit. Be sure to finish_writing and
finish_reading on the currently open request.

bzr 0.17.0 on python 2.4.4.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'push', 'bzr
+ssh://people.samba.org/data/bzr/jelmer/bzr/doc-update']

Some suggestions as to why this is happening would be nice.

Tags: hpss

Related branches

Revision history for this message
Jelmer Vernooij (jelmer) wrote : Re: [Bug 125784] Concurrency errors when using smart server over ssh

Remote has bzr 0.16.0

Revision history for this message
John A Meinel (jameinel) wrote : Re: Concurrency errors when using smart server over ssh

I think this actually happens any time we try to make a new request and the connection has closed.

I think the TooManyConcurrentRequests is actually the wrong exception to be raising. I don't know that for sure, but I'm guessing ConnectionClosed is the correct one.

Revision history for this message
Aaron Bentley (abentley) wrote :

This is a really bad error, and can be generated by using the Transport API correctly.

Changed in bzr:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Lucius (luciusf) wrote :

Also happening on Windows Server 2003 with version 0.9

Traceback (most recent call last):
  File "bzrlib\commands.pyc", line 817, in run_bzr_catch_errors
  File "bzrlib\commands.pyc", line 779, in run_bzr
  File "bzrlib\commands.pyc", line 477, in run_argv_aliases
  File "bzrlib\builtins.pyc", line 621, in run
  File "bzrlib\branch.pyc", line 137, in open_from_transport
  File "bzrlib\bzrdir.pyc", line 568, in open_from_transport
  File "bzrlib\transport\__init__.pyc", line 1505, in do_catching_redirections
  File "bzrlib\bzrdir.pyc", line 545, in find_format
  File "bzrlib\bzrdir.pyc", line 1249, in find_format
  File "bzrlib\bzrdir.pyc", line 2216, in probe_transport
  File "bzrlib\smart\medium.pyc", line 412, in get_request
  File "bzrlib\smart\medium.pyc", line 566, in __init__
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at
 0x0607D970>' has reached its concurrent request limit. Be sure to finish_writing and fini
sh_reading on the currently open request.

bzr 0.90.0 on python 2.5.1.final.0 (win32)

Revision history for this message
Adrian Wilkins (adrian-wilkins) wrote :

Another trace from version 1.1 on WinXP using the plain TCP bzr:// transport ; the target location is unreachable.

D:\src\vb6tools\ReleaseBuild>bzr pull bzr://192.168.5.128/c:/src/ReleaseBuild
bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartTCPClientMedium object at 0x0
1632670>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open r
equest.

Traceback (most recent call last):
  File "C:\Python25\Lib\site-packages\bzrlib\commands.py", line 806, in run_bzr_catch_errors
    return run_bzr(argv)
  File "C:\Python25\Lib\site-packages\bzrlib\commands.py", line 762, in run_bzr
    ret = run(*run_argv)
  File "C:\Python25\Lib\site-packages\bzrlib\commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "C:\Python25\Lib\site-packages\bzrlib\builtins.py", line 627, in run
    branch_from = Branch.open_from_transport(location_transport)
  File "C:\Python25\Lib\site-packages\bzrlib\branch.py", line 134, in open_from_transport
    control = bzrdir.BzrDir.open_from_transport(transport, _unsupported)
  File "C:\Python25\Lib\site-packages\bzrlib\bzrdir.py", line 688, in open_from_transport
    redirected)
  File "C:\Python25\Lib\site-packages\bzrlib\transport\__init__.py", line 1661, in do_catching_redirections
    return action(transport)
  File "C:\Python25\Lib\site-packages\bzrlib\bzrdir.py", line 665, in find_format
    transport, _server_formats=_server_formats)
  File "C:\Python25\Lib\site-packages\bzrlib\bzrdir.py", line 1417, in find_format
    return format.probe_transport(transport)
  File "C:\Python25\Lib\site-packages\bzrlib\bzrdir.py", line 2400, in probe_transport
    request = client.get_request()
  File "C:\Python25\Lib\site-packages\bzrlib\smart\medium.py", line 411, in get_request
    return SmartClientStreamMediumRequest(self)
  File "C:\Python25\Lib\site-packages\bzrlib\smart\medium.py", line 579, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartTCPClientMedium object at 0x01632670>' has reached its
concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

bzr 1.1.0 on python 2.5.1.final.0 (win32)
arguments: ['C:\\Python25\\Scripts\\bzr', 'pull', 'bzr://192.168.5.128/c:/src/ReleaseBuild']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  gtk d:\Documents and Settings\adwi2\Application Data\bazaar\2.0\plugins\gtk [0.94.0dev0]
  launchpad C:\Python25\lib\site-packages\bzrlib\plugins\launchpad [unknown]
  multiparent C:\Python25\lib\site-packages\bzrlib\plugins\multiparent.pyc [unknown]
  svn d:\Documents and Settings\adwi2\Application Data\bazaar\2.0\plugins\svn [0.4.8dev0]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Revision history for this message
Andrew Bennetts (spiv) wrote :

As reported in bug #144627 (and again recently on IRC) this can have the unfortunate effect of masking other exceptions. If an exception is raised within a method decorated by "@needs_write_lock", such as a KeyboardInterrupt or network error in branch.push, then when the decorator tries to unlock it will cause this error.

Possibly the needs_write_lock decorator should take care to always re-raise the original exception, and if another occurred during an unlock, to just emit a warning that bzr could not automatically unlock the object without discarding the original exception?

Revision history for this message
TPJ (tprimke) wrote : Re: TooManyConcurrentRequests when using smart server over ssh

I don't know wheter this information will be helpful, but I think I have encountered this bug too. My session:

$ bzr pull bzr+ssh://bazaar.launchpad.net/~tprimke/tputils-java/devel
Launchpad user 'tpj' doesn't have a registered SSH key
Permission denied (publickey).
bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0xb781b50c>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.
(...)

As a matter of fact I have updated the tputils-java/devel branch from other machine, and I don't have the private key on the machine I'm working right now (I don't know much about all those keys; I'm not even sure it matters).

I have tried it for 2 times: the first time with 0.90.0, and the second one with 1.3. The same bug.

Revision history for this message
Jamie Bennett (jamiebennett) wrote :
Download full text (4.3 KiB)

Another traceback if it helps

jamie@zander:~/develop/entertainer/bug-224691-2$ bzr push bzr+ssh://<email address hidden>/~jamie-bennett-gmail/entertainer/Bug-224691-2
/ [=== ] Transferring 0/4Connection to bazaar.launchpad.net closed by remote host.
bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0xb762b3ec>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 799, in run
    revision_id=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/bzrdir.py", line 204, in clone_on_transport
    revision_id=revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 127, in read_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 997, in clone
    self.copy_content_into(dest_repo, revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 907, in copy_content_into
    return InterRepository.get(self, destination).copy_content(revision_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 167, in write_locked
    self.unlock()
  File "/usr/lib/python2.5/site-packages/bzrlib/inter.py", line 110, in unlock
    self.target.unlock()
  File "/usr/lib/python2.5/site-packages/bzrlib/repository.py", line 983, in unlock
    self.control_files.unlock()
  File "/usr/lib/python2.5/site-packages/bzrlib/lockable_files.py", line 270, in unlock
    self._lock.unlock()
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 307, in unlock
    self.confirm()
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 395, in confirm
    info = self.peek()
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 418, in peek
    info = self._read_info_file(self._held_info_path)
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 408, in _read_info_file
    return self._parse_info(self.transport.get(path))
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/remote.py", line 184, in get
    return StringIO(self.get_bytes(relpath))
  File "/usr/lib/python2.5/site-packages/bzrlib/transport/remote.py", line 188, in get_bytes
    request = self.get_smart_medium().get_request()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 417, in get_request
    return SmartClientStreamMediumRequest(self)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/medium.py", line 592, in __init__
    raise errors.TooManyConcurrentRequests(self._medium)
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object ...

Read more...

Revision history for this message
Martin Pool (mbp) wrote :

So with Andrew's in-progress patch, you will at least see the real error that caused this. That might be e.g. the connection was closed by a network problem, or it might be an application or user error.

We'll still encounter an error in releasing the lock, in this case because the SmartClientMedium is in use by another request. We should probably at least print a warning about that.

However with this changed we'll still leave the remote repository locked, and it would be nice to avoid that. It seems like we should be able to ask the medium to abort or finish reading the response, and then we can go ahead and send the unlock.

Revision history for this message
Benjamin Peterson (benjaminp) wrote :

Robert Collins and I were talking about this on IRC. When I comment out the target.unlock in bzrlib.branch.BzrBranch.push, the pushing always work.

Revision history for this message
Benjamin Peterson (benjaminp) wrote :

Andrew's patch works for me! Can it be applied?

John A Meinel (jameinel)
Changed in bzr:
milestone: none → 1.6
status: Confirmed → Fix Committed
assignee: nobody → spiv
Revision history for this message
Benjamin Peterson (benjaminp) wrote :

I'm sorry to have had a false-positives, but the committed fix (I'm running off bzr.dev) doesn't fix the problem for me.

Changed in bzr:
status: Fix Committed → Confirmed
Revision history for this message
Andrew Bennetts (spiv) wrote :

Benjamin, could you provide a traceback from bzr.dev?

Revision history for this message
Benjamin Peterson (benjaminp) wrote :

Here it is. I should note this only happens reliably on commit to a remote branch now.

$ bzr ci -m "nothing"
Server does not understand Bazaar network protocol 3, reconnecting. (Upgrade the server to avoid this.)
Committing to: bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/
missing README
deleted README
[======================================================================================= ] Uploading data to master branch - Stage 4/6^bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x20755b0>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "/temp/bzr/bzr.dev/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/temp/bzr/bzr.dev/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/temp/bzr/bzr.dev/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/temp/bzr/bzr.dev/bzrlib/builtins.py", line 2382, in run
    author=author)
  File "/temp/bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/temp/bzr/bzr.dev/bzrlib/workingtree_4.py", line 241, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/temp/bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/temp/bzr/bzr.dev/bzrlib/mutabletree.py", line 197, in commit
    revprops=revprops, *args, **kwargs)
  File "/temp/bzr/bzr.dev/bzrlib/commit.py", line 404, in commit
    self._cleanup()
  File "/temp/bzr/bzr.dev/bzrlib/commit.py", line 588, in _cleanup
    raise e
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x20755b0>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

bzr 1.6b3 on python 2.5.2 (darwin)
arguments: ['/temp/bin/bzr', 'ci', '-m', 'nothing']
encoding: 'US-ASCII', fsenc: 'utf-8', lang: None
plugins:
  email /Users/erikpeterson/.bazaar/plugins/email [unknown]
  launchpad /temp/bzr/bzr.dev/bzrlib/plugins/launchpad [unknown]
  loom /Users/erikpeterson/.bazaar/plugins/loom [1.4.0dev0]

Revision history for this message
Andrew Bennetts (spiv) wrote :

Benjamin, that's actually a different bug. I've opened bug 243391 for it, let's continue talking about it there. I'll close and retitle this one to reduce confusion.

Revision history for this message
Andrew Bennetts (spiv) wrote :

This bug (TooManyConcurrentRequests during unlock) is fixed. Benjamin's error is bug 243391.

Changed in bzr:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.