lp-serve infinite loop on bzr branch/push development focus is empty bzrdir with "maximum recursion depth exceeded in __subclasscheck__"

[Andrew Bennetts]

This story starts nearly an hour ago, on #launchpad:

15:09 < Guest86981> I have a new Launchpad project and can't seem to push my bzr repository to my project on Launchpad. I get "Exception RuntimeError: 'maximum recursion depth exceeded in __subclasscheck__' in <type 'exceptions.RuntimeError'> ignored".

The command the user is trying is "bzr push lp:~m-os-griffin/mblogic/trunk"

Investigations showed that anyone trying to access the branch gets that problem. e.g. if I try to push to that branch, I get this -Dhpss trace:

"""
2.228 hpss: Built a new medium: SmartSSHClientMedium
2.229 hpss call: 'BzrDir.open_2.1', '~m-os-griffin/mblogic/trunk/'
2.229 (to bzr+ssh://bazaar.launchpad.net/~m-os-griffin/mblogic/trunk/)
2.260 ssh implementation is OpenSSH
16.819 result: ('yes', 'no')
16.819 hpss call: 'BzrDir.open_branchV3', '~m-os-griffin/mblogic/trunk/'
16.819 (to bzr+ssh://bazaar.launchpad.net/~m-os-griffin/mblogic/trunk/)
"""

Note that it is waiting for a reply to the open_branch call.

lp:~m-os-griffin/mblogic/trunk is just an empty bzrdir (no branch, no repository, just the bzr control dir):

"""
lftp bazaar.launchpad.net:~m-os-griffin/mblogic/trunk/.bzr> ls
-rw-r--r-- 1 1001 1001 141 Oct 28 06:24 README
-rw-r--r-- 1 1001 1001 35 Oct 28 06:24 branch-format
drwxr-xr-x 2 1001 1001 4096 Oct 28 06:24 branch-lock
"""

Also, that branch is the development focus for mblogic — perhaps the combination of a useless, empty bzrdir and being the dev focus causes this problem? Although the automatic stacking bzrdir doesn't seem to be present for this project.

This seems to be causing performance issues on codehosting, as the processes are stuck in an infinite loop that seems to gradually consume memory.

[Andrew Bennetts (spiv)]

I've suggested the user tries "bzr push sftp://bazaar.launchpad.net/~m-os-griffin/mblogic/trunk/ --use-existing-dir" as a workaround; if it does work it may resolve this instance of the problem. If my guess about dev focus + empty bzrdir is correct we'll be able to reproduce though.

FWIW, accessing over SFTP appears normal, the client doesn't get stuck in an infinite loop or anything like that:

$ bzr revno sftp://bazaar.launchpad.net/~m-os-griffin/mblogic/trunk
bzr: ERROR: Not a branch: "sftp://bazaar.launchpad.net/~m-os-griffin/mblogic/trunk/.bzr/branch/"

[Andrew Bennetts (spiv)]

From email with the IRC reporter (reproduced with permission):

"""
I have an existing project called MBLogic. This is a combination of
Python, Javascript, xHTML, and SVG. This was *not* previously in bzr.

1) I copied the project to a directory called
~/Projects/mblogic/trunk. This was from the current release (I have
been releasing tar balls on Sourceforge).

2) I then did the following:

bzr init
bzr add
bzr commit -m "Initial import"

3) Next, I branched the project to a new branch called "formconfig".

bzr branch trunk formconfig
cd formconfig
(add, delete, and touch the new files - The touch was necessary for
the new files as they already existed before the branch was made.)
bzr add
bzr commit -m "Description of new features"

4) Next I merged the changes back into trunk.
cd ../trunk
bzr merge ../formconfig
bzr commit -m "Form based configuration (etc.)"

5) Then I tried pushing to my project on Launchpad.

bzr launchpad-login m-os-griffin
bzr push lp:~m-os-griffin/mblogic/trunk

I tried this both with and without trying "bzr launchpad-login
m-os-griffin". It doesn't seem to make any difference.

I tried this yesterday with the same results. I waited 24 hours and
tried again (in case it was just a server problem). Disregard anything
that I said on IRC about pushing non-existent projects. That was just
me not reading the log correctly.

If I find out anything new, I will contact you again.
"""

At this stage I don't think there's anything especially relevant to this bug in that description, but I'm putting it here in case I'm wrong.

[Andrew Bennetts (spiv)]

I think this may be fixed by r5102 of <lp:bzr/2.2>, which will be part of 2.2.2 when it is released.

[Paul Sladen (sladen)]

I've just hit this now (duped as bug #671351). Could somebody confirm+bump the priority up.

[Paul Sladen (sladen)]

rhlee has confirmed this too.

[Paul Sladen (sladen)]

Para-phrasing of spiv's useful answers on IRC about what needs doing.

I've fixed the bzr side in lp:bzr/2.2 to be more robust (i.e. not get stuck in an infinite loop of error handling callbacks). The update needs to be installed by the launchpad-code team; thumper is the lead there
it got past the unit tests because it involves a pretty unusual scenario that triggers an unexpected interaction between codehosting and bzrlib

IIRC it's basically that some sort broken/unusual bzrdir on the remote side can cause codehosting's VFS to raise a slightly odd error (from bzrlib's perspective at least) in response to an open_branch request from the client. Arguably codehosting's VFS should be more careful to only raise filesystem errors from its functions, and arguably bzr should be more robust against unexpected errors.

You can probably workaround it by using sftp or nosmart+lp URLs.

[Paul Sladen (sladen)]

I've tried all sorts of combinations of --overwrite, --use-existing-dir, nosmart+lp:ubuntu-font-family-website and sftp:// URIs, which returns variations on:

  bzr push --overwrite sftp://<email address hidden>/~uffw-contributors/ubuntu-font-family-website/trunk
  bzr: ERROR: At sftp://<email address hidden>/~uffw-contributors/ubuntu-font-family-website/trunk you have a valid .bzr control directory, but not a branch or repository. This is an unsupported configuration. Please move the target directory out of the way and try again.

so something seems _broken_ enough that just trying to ovewrite more --forcefully doesn't work.

[Andrew Bennetts (spiv)]

bzr 2.2.2 has been deployed, and so has an extra fix to avoid calling NotBranchError._format method entirely from its repr. We think it's fixed now. Obviously feel free to reopen or file a new bug if this seems to happen again.

[Andrew Bennetts (spiv)]

As of r5115 of lp:bzr/2.2 we're pretty sure this is fixed.