BZR crashes when large amounts of files are committed through TortoiseBZR over bzr+ssh from 2.3.1 client to 2.2.1 server.

[ced]

After gathering all the files, the app gives a crash report:
    Connected (version 2.0, client OpenSSH_5.5p1)
    bzr: ERROR: bzrlib.errors.TooManyConcurrentRequests: The medium 'SmartSSHClientMedium(bzr+ssh://bzr@leandra/)' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

    Traceback (most recent call last):
      File "bzrlib\commands.pyo", line 926, in exception_to_return_code
      File "bzrlib\commands.pyo", line 1126, in run_bzr
      File "bzrlib\commands.pyo", line 691, in run_argv_aliases
      File "bzrlib\commands.pyo", line 713, in run
      File "bzrlib\cleanup.pyo", line 135, in run_simple
      File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
      File "C:/Program Files (x86)/Bazaar/plugins\qbzr\lib\commands.py", line 788, in run
      File "C:/Program Files (x86)/Bazaar/plugins\qbzr\lib\subprocess.py", line 888, in run_subprocess_command
      File "bzrlib\commands.pyo", line 1126, in run_bzr
      File "bzrlib\commands.pyo", line 691, in run_argv_aliases
      File "bzrlib\commands.pyo", line 713, in run
      File "bzrlib\cleanup.pyo", line 135, in run_simple
      File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
      File "bzrlib\builtins.pyo", line 3241, in run
      File "bzrlib\decorators.pyo", line 194, in write_locked
      File "bzrlib\workingtree_4.pyo", line 197, in commit
      File "bzrlib\decorators.pyo", line 194, in write_locked
      File "bzrlib\mutabletree.pyo", line 201, in commit
      File "bzrlib\commit.pyo", line 286, in commit
      File "bzrlib\cleanup.pyo", line 131, in run
      File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
      File "bzrlib\commit.pyo", line 360, in _commit
      File "bzrlib\commit.pyo", line 512, in _check_bound_branch
      File "bzrlib\decorators.pyo", line 140, in read_locked
      File "bzrlib\branch.pyo", line 2737, in get_master_branch
      File "bzrlib\branch.pyo", line 176, in open
      File "bzrlib\bzrdir.pyo", line 717, in open
      File "bzrlib\bzrdir.pyo", line 752, in open_from_transport
      File "bzrlib\bzrdir.pyo", line 1699, in open
      File "bzrlib\bzrdir.pyo", line 2886, in _open
      File "bzrlib\remote.pyo", line 120, in __init__
      File "bzrlib\remote.pyo", line 132, in _probe_bzrdir
      File "bzrlib\remote.pyo", line 139, in _rpc_open_2_1
      File "bzrlib\remote.pyo", line 57, in _call
      File "bzrlib\smart\client.pyo", line 132, in call
      File "bzrlib\smart\client.pyo", line 145, in call_expecting_body
      File "bzrlib\smart\client.pyo", line 90, in _call_and_read_response
      File "bzrlib\smart\client.pyo", line 45, in _send_request
      File "bzrlib\smart\client.pyo", line 115, in _construct_protocol
      File "bzrlib\smart\medium.pyo", line 711, in get_request
      File "bzrlib\smart\medium.pyo", line 969, in __init__
    TooManyConcurrentRequests: The medium 'SmartSSHClientMedium(bzr+ssh://bzr@leandra/)' has reached its concurrent request limit. Be sure to finish_writing and finish_reading on the currently open request.

    bzr 2.3.1 on python 2.6.6 (Windows-7-6.1.7601-SP1)
    arguments: ['C:\\Program Files (x86)\\Bazaar\\bzr.exe', 'qsubprocess', '--bencode', '@c:/users/ced/appdata/local/temp/QBzr/qsubprocess/tmp_6gh4q']
    encoding: 'cp1252', fsenc: 'mbcs', lang: None
    plugins:
      bzrtools C:\Program Files (x86)\Bazaar\plugins\bzrtools [2.3.1]
      colo C:\Program Files (x86)\Bazaar\plugins\colo [0.2.1]
      explorer C:\Program Files (x86)\Bazaar\plugins\explorer [1.1.2]
      fastimport C:\Program Files (x86)\Bazaar\plugins\fastimport [0.10.0]
      launchpad C:\Program Files (x86)\Bazaar\plugins\launchpad [2.3.1]
      loom C:\Program Files (x86)\Bazaar\plugins\loom [2.2.1dev]
      netrc_credential_store C:\Program Files (x86)\Bazaar\plugins\netrc_credential_store [2.3.1]
      news_merge C:\Program Files (x86)\Bazaar\plugins\news_merge [2.3.1]
      pipeline C:\Program Files (x86)\Bazaar\plugins\pipeline [1.1.0]
      qbzr C:\Program Files (x86)\Bazaar\plugins\qbzr [0.20.0]
      rewrite C:\Program Files (x86)\Bazaar\plugins\rewrite [0.6.2dev]
      svn C:\Program Files (x86)\Bazaar\plugins\svn [1.0.5dev]
      upload C:\Program Files (x86)\Bazaar\plugins\upload [1.0.0]
      xmloutput C:\Program Files (x86)\Bazaar\plugins\xmloutput [0.8.7.dev]

    *** Bazaar has encountered an internal error. This probably indicates a
        bug in Bazaar. You can help us fix it by filing a bug report at
            https://bugs.launchpad.net/bzr/+filebug
        including this traceback and a description of the problem.

This only happens with commits that hold large numbers of files, in this case about 10,000 files with a total of about 70MBytes of data.
'Smaller' commits, e.g 720MB but only 250 files worked fine on the exact same repository, same protocol, etcetera.

Note that this problem ONLY occurs when committing from a 2.3.1 client to a 2.2.1 server!
I solved this after i installed 2.3.1 on the server, however this might still be worth reporting since not all servers can be upgraded all the time.

[Alexander Belchenko (bialix)]

Can you show us the relevant part of the .bzr.log for failing coimmit?
How long the failing operation takes? Maybe it's related to the TBZR files scanner that runs while commit still in the progress. Can you disable TortoiseBzr and try to commit the same files from Bazaar Explorer, or even from command-line with `bzr qcommit` command?

[ced (alexanderypema)]

@Alexander That'd be hard to reproduce right now, since i already upgraded the server..
Also, the .bzr.log file contents are identical to the error thrown i posted here.

[ced (alexanderypema)]

@Alexander Although, i suppose all you'd have to do, is setup a 2.2.1 server (current version in ubuntu default repositories), a 2.3.1 client, and submit a lot of files (~5000 would do probably) using the bzr+ssh protocol

[Andrew Bennetts (spiv)]

As this is a bug in the client, it's odd that upgrading the server should have affected it, especially as I don't think we added any new smart protocol verbs in 2.3.

Given that this is happening on the initial request to open a branch, the most obvious cause of this bug would be reusing a connection that's already being used. Looking over the code MutableTree.commit initialises a fresh list for “possible_master_transports”, and the only thing that happens that might populate that list before that traceback is a call to commit.Commit.update_revprops. That in some circumstances will call get_master_branch, and has this nasty except block around that call:

            except errors.BzrError, e:
                # Silently fall back to local implicit nick if the master is
                # unavailable
                mutter("Could not connect to bound branch, "
                    "falling back to local nick.\n " + str(e))

If for some reason get_master_branch failed, maybe the connection failed or was interrupted, or maybe even just something more mundane like a PermissionDenied (although I don't think so), maybe this would leave the smart medium in a poor state and silently swallow that error.

I don't see any other likely explanations for how this could have occurred.

It would be good to see the .bzr.log for that failing commit to confirm that muttered message is present, and see what error is being suppressed here.

Probably we should just stop catching all BzrErrors here, or avoid adding a failed connection to possible_transports, or something like that.

[Andrew Bennetts (spiv)]

If my theory is correct, a workaround may be to set an explicit branch nick in your checkout so it doesn't have to open the master to find it. Although if get_master_branch is failing then I'd still expect the commit to fail a bit further on, but it would then fail with a more relevant error.

[Andrew Bennetts (spiv)]

Although we don't have enough details to know what went wrong specifically, there's enough of a likely suspect Branch._get_nick's exception handling that I'm going to mark this Confirmed rather than Incomplete, so we don't scare people off working on this. (Certainly improving that code would be a worthwhile improvement even if it turns out the reporter's problem is something else.)

The full .bzr.log output from the crash would confirm my diagnosis, though.

[ced (alexanderypema)]

Log from bzr.log.old:

Wed 2011-03-23 13:55:38 +0100
0.139 bazaar version: 2.3.1
0.139 bzr arguments: [u'qsubprocess', u'--bencode', u'@c:/users/ced/appdata/local/temp/QBzr/qsubprocess/tmpbvu5kj']
0.208 looking for plugins in C:/Users/Ced/AppData/Roaming/bazaar/2.0/plugins
0.208 looking for plugins in C:/Program Files (x86)/Bazaar/plugins
0.559 encoding stdout as osutils.get_user_encoding() 'cp1252'
0.791 bazaar version: 2.3.1
0.860 bzr arguments: [u'commit', u'-m', u'Initial checkins, shitloads of data. Some are corrupt, but will be fixed later.', u'website related/New folder',.........[cut out the actual file list here, since it's huge and not really relevant]
1.235 encoding stdout as osutils.get_user_encoding() 'cp1252'
1.331 opening working tree 'D:/Project files'
196.755 bzr-svn: using Subversion 1.6.6 (), subvertpy 0.7.5
197.097 falling back to default implementation
197.097 failed to load system host keys: [Errno 2] No such file or directory: 'C:\\Users\\Ced/.ssh/known_hosts'
[ 3756] 2011-03-23 13:58:56.084 INFO: Connected (version 2.0, client OpenSSH_5.5p1)
451.629 Could not connect to bound branch, falling back to local nick.
 Unable to connect to target of bound branch BzrBranch7(file:///D:/Project%20files/) => bzr+ssh://bzr@leandra/~/programming/: Connection error: Unable to authenticate to SSH host as
  bzr@leandra
 No existing session
451.632 preparing to commit
451.995 Transferred: 0kB (0.0kB/s r:0kB w:0kB)
451.995 Transferred: 0kB (0.0kB/s r:0kB w:0kB)
451.996 Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 926, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1126, in run_bzr
  File "bzrlib\commands.pyo", line 691, in run_argv_aliases
  File "bzrlib\commands.pyo", line 713, in run
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "C:/Program Files (x86)/Bazaar/plugins\qbzr\lib\commands.py", line 788, in run
  File "C:/Program Files (x86)/Bazaar/plugins\qbzr\lib\subprocess.py", line 888, in run_subprocess_command
  File "bzrlib\commands.pyo", line 1126, in run_bzr
  File "bzrlib\commands.pyo", line 691, in run_argv_aliases
  File "bzrlib\commands.pyo", line 713, in run
  File "bzrlib\cleanup.pyo", line 135, in run_simple
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 3241, in run
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\workingtree_4.pyo", line 197, in commit
  File "bzrlib\decorators.pyo", line 194, in write_locked
  File "bzrlib\mutabletree.pyo", line 201, in commit
  File "bzrlib\commit.pyo", line 286, in commit
  File "bzrlib\cleanup.pyo", line 131, in run
  File "bzrlib\cleanup.pyo", line 165, in _do_with_cleanups
  File "bzrlib\commit.pyo", line 360, in _commit
  File "bzrlib\commit.pyo", line 512, in _check_bound_branch
  File "bzrlib\decorators.pyo", line 140, in read_locked
  File "bzrlib\branch.pyo", line 2737, in get_master_branch
  File "bzrlib\branch.pyo", line 176, in open
  File "bzrlib\bzrdir.pyo", line 717, in open
  File "bzrlib\bzrdir.pyo", line 752, in open_from_transport
  File "bzrlib\bzrdir.pyo", line 169...

[Andrew Bennetts (spiv)]

Great, these lines confirm my theory:

451.629 Could not connect to bound branch, falling back to local nick.
 Unable to connect to target of bound branch BzrBranch7(file:///D:/Project%20files/) => bzr+ssh://bzr@leandra/~/programming/: Connection error: Unable to authenticate to SSH host
as
  bzr@leandra
 No existing session