TooManyConcurrentRequests during commit

[Andrew Bennetts]

This was initially reported by Benjamin Peterson in <https://bugs.launchpad.net/bzr/+bug/125784/comments/14>:

"""
Here it is. I should note this only happens reliably on commit to a
remote branch now.

$ bzr ci -m "nothing"
Server does not understand Bazaar network protocol 3, reconnecting. (Upgrade the server to avoid this.)
Committing to: bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/
missing README
deleted README
[======================================================================================= ] Uploading data to master branch - Stage 4/6^bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium
+'<bzrlib.smart.medium.SmartSSHClientMedium object at 0x20755b0>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

Traceback (most recent call last):
  File "/temp/bzr/bzr.dev/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/temp/bzr/bzr.dev/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/temp/bzr/bzr.dev/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/temp/bzr/bzr.dev/bzrlib/builtins.py", line 2382, in run
    author=author)
  File "/temp/bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/temp/bzr/bzr.dev/bzrlib/workingtree_4.py", line 241, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/temp/bzr/bzr.dev/bzrlib/decorators.py", line 192, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/temp/bzr/bzr.dev/bzrlib/mutabletree.py", line 197, in commit
    revprops=revprops, *args, **kwargs)
  File "/temp/bzr/bzr.dev/bzrlib/commit.py", line 404, in commit
    self._cleanup()
  File "/temp/bzr/bzr.dev/bzrlib/commit.py", line 588, in _cleanup
    raise e
TooManyConcurrentRequests: The medium '<bzrlib.smart.medium.SmartSSHClientMedium object at 0x20755b0>' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

bzr 1.6b3 on python 2.5.2 (darwin)
arguments: ['/temp/bin/bzr', 'ci', '-m', 'nothing']
encoding: 'US-ASCII', fsenc: 'utf-8', lang: None
plugins:
  email /Users/erikpeterson/.bazaar/plugins/email [unknown]
  launchpad /temp/bzr/bzr.dev/bzrlib/plugins/launchpad [unknown]
  loom /Users/erikpeterson/.bazaar/plugins/loom [1.4.0dev0]
"""

The problem seems to be that bzrlib.commit.Commit._cleanup isn't handling errors carefully enough. I think it needs to behave more like the @needs_write_lock decorator: capture the original sys.exc_info() and re-raise *that* rather than subsequent errors.

[Andrew Bennetts (spiv)]

Benjamin, could you please reproduce the TooManyConcurrentRequests with this command:

   bzr -Dhpss ci -m "nothing"

And attach the resulting ~/.bzr.log output to this bug? Thanks!

[Benjamin Peterson (benjaminp)]

Here's all the gory details:

0.589 encoding stdout as sys.stdout encoding 'US-ASCII'
0.591 bzr arguments: [u'-Dhpss', u'ci', u'-m', u'even less']
0.591 looking for plugins in /Users/erikpeterson/.bazaar/plugins
0.801 looking for plugins in /temp/bzr/bzr.dev/bzrlib/plugins
0.801 Plugin name __init__ already loaded
0.801 Plugin name __init__ already loaded
0.809 looking for plugins in /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/site-packages/bzrlib/plugins
0.819 encoding stdout as sys.stdout encoding 'US-ASCII'
1.053 opening working tree '/temp/python/bzr/sample'
1.100 preparing to commit
1.144 hpss: Built a new medium: SmartSSHClientMedium
1.172 hpss call: 'BzrDir.open', 'python/users/benjamin.peterson/sample/'
1.172 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
1.203 ssh implementation is OpenSSH
[ 1415] 2008-06-26 22:31:32.872 WARNING: Server does not understand Bazaar network protocol 3, reconnecting. (Upgrade the server to avoid this.)
4.985 hpss call: 'BzrDir.open', 'python/users/benjamin.peterson/sample/'
4.986 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
16.465 result: 11.478s 'yes',
16.476 hpss call: 'BzrDir.open_branch', 'python/users/benjamin.peterson/sample/'
16.477 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
16.670 result: 0.193s 'ok', ''
16.671 hpss call: 'BzrDir.find_repositoryV2', 'python/users/benjamin.peterson/sample/'
16.671 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
16.859 result: 0.188s 'error', "Generic bzr smart protocol error: bad request 'BzrDir.find_repositoryV2'"
16.859 hpss call: 'BzrDir.find_repository', 'python/users/benjamin.peterson/sample/'
16.860 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
17.103 result: 0.243s 'ok', '../../..', 'yes', 'no'
17.113 hpss call: 'BzrDir.open', 'python/users/benjamin.peterson/'
17.113 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
17.293 result: 0.179s 'no',
17.296 hpss call: 'BzrDir.open', 'python/users/'
17.296 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
17.485 result: 0.189s 'no',
17.488 hpss call: 'BzrDir.open', 'python/'
17.489 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
17.662 result: 0.173s 'yes',
17.664 hpss call: 'BzrDir.find_repositoryV2', 'python/'
17.664 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
17.842 result: 0.178s 'error', "Generic bzr smart protocol error: bad request 'BzrDir.find_repositoryV2'"
17.842 hpss call: 'BzrDir.find_repository', 'python/'
17.842 (to bzr+ssh://<email address hidden>/python/users/benjamin.peterson/sample/)
18.026 result: 0.184s 'ok', '', 'yes', 'no'
18.028 hpss call: 'Repository.is_shared', 'python/'
18.028 (to bzr+ssh://<email address hidden>...

[Benjamin Peterson (benjaminp)]

Note the above log was obtained when the commit seemed to hang and I gave a KeyboardInterrupt. (Could the TooManyConcurrentRequests be masking that?) If I wait ~7 minutes, the commit will succeed and I'm attaching the log for that case.

[Andrew Bennetts (spiv)]

(The attached log isn't actually complete. It's only the last 80 seconds or so of the commit, interspersed with a "bzr status macostools.py" that you must have done during the commit. The log needs to go back to a line like the "0.591 bzr arguments: [u'-Dhpss', u'ci', u'-m', u'even less']" line in your earlier log.)

You're right, the KeyboardInterrupt is probably being masked by the TooManyConcurrentRequests. We should find out why KeyboardInterrupt during commit can cause that.

Perhaps the code that raises TooManyConcurrentRequests should first check "if sys.exc_info() != (None, None, None)", i.e. if there is already a current exception, and if so write it to the log file before raising a new exception.

[Benjamin Peterson (benjaminp)]

Ok. I hope the attached log here helps.

[GadAbraham (gad-abraham)]

I have the same TooManyConcurrentRequests problem, with bzr 1.3.1 on Hardy committing to another machine (also 1.3.1 on Hardy). I don't think Benjamin's log shows the problem (it exits with status 0), so here's mine.

[Andrew Bennetts (spiv)]

GadAbraham: Thanks for that log, that's quite helpful. It appears that Commit._cleanup is the culprit. I think it needs to behave more like the @needs_write_lock decorator in how it handles errors during cleanup.

[Andrew Bennetts (spiv)]

I intend to fix this for 1.8. It's not too hard to fix, and is needlessly frustrating for a user.

[prajwala (prajwala)]

I have the same problem. Now I am unable to push or merge to the parent branch. How to go ahead. I mean what to do now to continue my work.

[Andrew Bennetts (spiv)]

I've submitted a patch for this to the mailing list (see the attachment on https://lists.ubuntu.com/archives/bazaar/2008q4/048296.html).

[Alwyn Schoeman (alwyn-schoeman)]

I am still getting this using version 1.10 client on Cygwin. I'm connecting to a server on version 0.9.

Was this ever integrated?

[Martin Pool (mbp)]

2009/1/19 Alwyn Schoeman <email address hidden>:
> I am still getting this using version 1.10 client on Cygwin. I'm
> connecting to a server on version 0.9.
>
> Was this ever integrated?

Hi Alwyn,

Because the TooManyConcurrentRequests is masking an underlying error,
you may be seeing a different problem. Could you please file a new
bug, including the .bzr.log output from running a command like

   bzr -Dhpss pull

(or whatever specific operation is failing for you.)

--
Martin <http://launchpad.net/~mbp/>

[Vincent Ladeuil (vila)]

Reverting status to 'In progress' as I can't find any branch where this is being addressed (I'm sure some branch exists though, it just seems it's not public :).

Andrew, can you give some feedback on the subject and update (or clear) the milestone ?

[Robert Collins (lifeless)]

 status fix-committed

This was fixed and commited.

Alwyn, please file a new bug for the problem you are experiencing.

TooManyConcurrentRequests can have many causes. Its like 'SIGSEGV' in a
C program.

-Rob

[Robert Collins (lifeless)]

status fixcommitted

This was fixed and commited.

Alwyn, please file a new bug for the problem you are experiencing.

TooManyConcurrentRequests can have many causes. Its like 'SIGSEGV' in a
C program.

-Rob