failure to push to Launchpad 'branchlock'

[John A Meinel]

I just updated to bzr.dev 6345 today and I started getting failures trying to do just a plain "bzr push" to update one of my branches. I'll include the traceback.

The weirdest part is the failure is:
LockFailed: Cannot lock LockDir(lp-84933520:///~jameinel/u1db/index-transformations/.bzr/branchlock): Transport operation not possible: readonly transport

Now, this is pushing to my ~jameinel branch so it shouldn't be a readonly issue. Further, it is saying it cannot lock .../.bzr/branchlock which isn't a directory we should be using, instead we should be using .bzr/branch/lock.

My guess is that we're missing a trailing slash somewhere on .bzr/branch/ when combining to create .bzr/branch/lock url.

I tried doing "bzr revert -r 6145" and it still failed. On IRC mgz mentioned revision 6079 as being known-buggy, so I reverted back to "bzr revert -r 6078" and that push did, indeed, work.

I tried figuring out what revision of bzr.dev I was using previously, but we don't seem to log that yet. "history-db" seems to indicate I added 24 mainline revs, 101 total revs, which would be consistent with updating from ~revno 6322 up to 6345. (bzr log -n1 -r-24 | grep revno shows 101 revisions added)

3.924 result: ('yes', 'no')
3.924 hpss call: 'BzrDir.open_branchV3', '~jameinel/u1db/index-transformations/'
3.924 (to bzr+ssh://bazaar.launchpad.net/~jameinel/u1db/index-transformations/)
4.051 result: ('branch', 'Bazaar Branch Format 7 (needs bzr 1.6)\n')
4.051 hpss call: 'BzrDir.find_repositoryV3', '~jameinel/u1db/index-transformations/'
4.051 (to bzr+ssh://bazaar.launchpad.net/~jameinel/u1db/index-transformations/)
4.106 result: ('ok', '', 'yes', 'no', 'yes', 'Bazaar repository format 2a (needs bzr 1.16 or later)\n')
4.109 hpss call: 'Branch.get_stacked_on_url', '~jameinel/u1db/index-transformations/'
4.109 (to bzr+ssh://bazaar.launchpad.net/~jameinel/u1db/index-transformations/)
4.170 result: ('ok', '/+branch-id/511801')
4.171 hpss call: 'BzrDir.open_2.1', '+branch-id/511801/'
4.171 (to bzr+ssh://bazaar.launchpad.net/~jameinel/u1db/index-transformations/)
4.294 result: ('yes', 'no')
4.294 hpss call: 'BzrDir.open_branchV3', '+branch-id/511801/'
4.294 (to bzr+ssh://bazaar.launchpad.net/~jameinel/u1db/index-transformations/)
4.352 result: ('branch', 'Bazaar Branch Format 7 (needs bzr 1.6)\n')
4.354 hpss call: 'BzrDir.find_repositoryV3', '+branch-id/511801/'
4.354 (to bzr+ssh://bazaar.launchpad.net/~jameinel/u1db/index-transformations/)
4.418 result: ('ok', '', 'yes', 'no', 'yes', 'Bazaar repository format 2a (needs bzr 1.16 or later)\n')
4.422 hpss call: 'Branch.get_stacked_on_url', '+branch-id/511801/'
4.422 (to bzr+ssh://bazaar.launchpad.net/~jameinel/u1db/index-transformations/)
4.487 result: ('NotStacked',)
4.519 hpss call: 'Branch.lock_write', '~jameinel/u1db/index-transformations/', '', ''
4.520 (to bzr+ssh://bazaar.launchpad.net/~jameinel/u1db/index-transformations/)
4.607 result: ('LockFailed', 'LockDir(lp-84933520:///~jameinel/u1db/index-transformations/.bzr/branchlock)', 'Transport operation not possible: readonly transport ')
4.608 Transferred: 2kB (0.7kB/s r:1kB w:1kB)
4.628 Traceback (most recent call last):
  File "C:\dev\bzr\bzr.dev\bzrlib\commands.py", line 923, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "C:\dev\bzr\bzr.dev\bzrlib\commands.py", line 1128, in run_bzr
    ret = run(*run_argv)
  File "C:\dev\bzr\bzr.dev\bzrlib\commands.py", line 676, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "C:\dev\bzr\bzr.dev\bzrlib\commands.py", line 698, in run
    return self._operation.run_simple(*args, **kwargs)
  File "C:\dev\bzr\bzr.dev\bzrlib\cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "C:\dev\bzr\bzr.dev\bzrlib\cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "C:\dev\bzr\bzr.dev\bzrlib\builtins.py", line 1238, in run
    use_existing_dir=use_existing_dir, no_tree=no_tree)
  File "C:\dev\bzr\bzr.dev\bzrlib\push.py", line 143, in _show_push_branch
    remember, create_prefix)
  File "C:\dev\bzr\bzr.dev\bzrlib\controldir.py", line 403, in push_branch
    overwrite, stop_revision=revision_id)
  File "C:\dev\bzr\bzr.dev\bzrlib\branch.py", line 1140, in push
    lossy, *args, **kwargs)
  File "C:\dev\bzr\bzr.dev\bzrlib\branch.py", line 3472, in push
    op.add_cleanup(self.target.lock_write().unlock)
  File "C:\dev\bzr\bzr.dev\bzrlib\remote.py", line 3294, in lock_write
    remote_tokens = self._remote_lock_write(token)
  File "C:\dev\bzr\bzr.dev\bzrlib\remote.py", line 3278, in _remote_lock_write
    repo_token or '', **err_context)
  File "C:\dev\bzr\bzr.dev\bzrlib\remote.py", line 70, in _call
    self._translate_error(err, **err_context)
  File "C:\dev\bzr\bzr.dev\bzrlib\remote.py", line 3127, in _translate_error
    self.repository._translate_error(err, branch=self, **context)
  File "C:\dev\bzr\bzr.dev\bzrlib\remote.py", line 1250, in _translate_error
    self.bzrdir._translate_error(err, repository=self, **context)
  File "C:\dev\bzr\bzr.dev\bzrlib\remote.py", line 484, in _translate_error
    _translate_error(err, bzrdir=self, **context)
  File "C:\dev\bzr\bzr.dev\bzrlib\remote.py", line 3879, in _translate_error
    raise translator(err)
LockFailed: Cannot lock LockDir(lp-84933520:///~jameinel/u1db/index-transformations/.bzr/branchlock): Transport operation not possible: readonly transport

4.630 return code 3
[ 6560] 2011-12-06 14:59:13.571 INFO: HPSS calls: 9 (0 vfs) SmartSSHClientMedium(bzr+ssh://<email address hidden>/)

[John A Meinel (jameinel)]

Note, I was able to reproduce this several times until I finally succeeded with bzr.dev 6078, and now I cannot reproduce with the same branch.

[John A Meinel (jameinel)]

I did get these failure emails from Launchpad, perhaps relevant:

Launchpad encountered an internal error during the following operation: generating an incremental diff for a merge proposal. It was logged with id OOPS-34715413953d196dcd32554b5f72d17f. Sorry for the inconvenience.

Launchpad encountered an internal error during the following operation: generating the diff for a merge proposal. It was logged with id OOPS-1d3b3ce27d7feb55c0b4d16df78c6ab0. Sorry for the inconvenience.

Launchpad encountered an internal error during the following operation: generating an incremental diff for a merge proposal. It was logged with id OOPS-42d288738b05c362e0ca542335d012ce. Sorry for the inconvenience.

Launchpad encountered an internal error during the following operation: generating the diff for a merge proposal. It was logged with id OOPS-0698def6a331c4cedf0d6708b4e6d764. Sorry for the inconvenience.

[John A Meinel (jameinel)]

Given the simultaneous errors from Launchpad, I'm wondering if there was a problem with the codehosting machine.

[Gavin Panella (allenap)]

The librarian is broken at the moment and the diff generating code is unable to upload.

[Julian Edwards (julian-edwards)]

This bug is invalid, there's no code issue here.

[Martin Packman (gz)]

As best as I could tell on the day, this was somehow caused by the issues launchpad was having at the time, and though there's probably a formatting bug with the remote branch lock as well, I had no luck reproducing so don't know who's to blame for losing the slash.