flash-kernel import fails apparently mismatching maverick and natty branches

[Loïc Minier]

Hi

flash-kernel was developed in a regular bzr branch which we used to overwrite the UDD branch

This then required a requeue_package.py --full flash-kernel, but this failed and currently still fails with:
Traceback (most recent call last):
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1095, in <module>
    persistent_download_cache=options.persistent_download_cache))
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 1021, in main
    import_package(temp_dir, download_dir, importp, revid_db, bstore, possible_transports=possible_transports)
  File "/srv/package-import.canonical.com/new/scripts/import_package.py", line 541, in import_package
    possible_transports=possible_transports)
  File "/usr/lib/python2.5/site-packages/bzrlib/controldir.py", line 374, in sprout
    create_tree_if_local=create_tree_if_local)
  File "/usr/lib/python2.5/site-packages/bzrlib/cleanup.py", line 131, in run
    self.cleanups, self.func, self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/controldir.py", line 436, in _sprout
    repository=result_repo)
  File "<string>", line 4, in sprout_read_locked
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 1289, in sprout
    result = to_bzrdir.create_branch(repository=repository)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 254, in create_branch
    name=name, repository=repository)
  File "/usr/lib/python2.5/site-packages/bzrlib/remote.py", line 2143, in initialize
    % (repository.user_url, medium.base, repo_path))
AssertionError: repository.user_url 'bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/maverick/flash-kernel/maverick-201102281451/' does not match URL from server response ('bzr+ssh://bazaar.launchpad.net/~ubuntu-branches/ubuntu/natty/flash-kernel/natty/' + '')

The maverick vs natty part is suspicious in this log.

Thanks,

[James Westby (james-w)]

I think this might be something to do with reusing transports.

We are attempting to push a branch to the maverick path that we just branched from the natty
path.

Thanks,

James

[Andrew Bennetts (spiv)]

It would be helpful to see an HPSS trace of this. (In generaly add -Dhpss to the command line, then attach the resulting ~/.bzr.log section to this bug, but I think the package importer needs different magic?)

That check is new, and I haven't seen it trip before. The check is part of creating a new branch if:
 * the client already knows which repository the branch should use before it creates it (and it does in this case, probably because it just created it!), and
 * when we create the branch the server says it is using a different repository for that branch!

This situation was possible but undetected in earlier bzr's. I added the check because bzr now avoids reopening the repository after creating the branch, and so this check is a sanity check that that optimisation is valid.

One way this could fail on Launchpad is a race with a concurrent branch rename, but that doesn't seem likely for a newly created branch. (And aborting rather than continuing in that situation is probably a reasonable thing for bzr to do.)

Launchpad doesn't use shared repositories so it really should be the case that the repository user_url matches the branch user_url (modulo rename races).

I think James' guess that it's to do with reusing transports may well be right: it looks like that code may be accidentally interpreting the path from the server relative to the transport's base path rather than the request's base path. I'll make a patch.

[Andrew Bennetts (spiv)]

Fixed in lp:bzr/2.3 r5624 and lp:bzr r5696.

[James Westby (james-w)]

Thanks Andrew. We should get an updated bzr on Jubany, as a fair
number of packages are hitting this, and it happens when there is
a collision, which leaves people open to starting from the wrong
code if they grab the branch.

Would you file an RT ticket for that please?

Thanks,

James

On Thu, 03 Mar 2011 08:55:40 -0000, Andrew Bennetts <email address hidden> wrote:
> Fixed in lp:bzr/2.3 r5624 and lp:bzr r5696.
>
> ** Changed in: bzr
> Status: Confirmed => Fix Released
>
> --
> You received this bug notification because you are a member of bzr-core,
> which is subscribed to Bazaar.
> https://bugs.launchpad.net/bugs/726584
>
> Title:
> flash-kernel import fails apparently mismatching maverick and natty
> branches
>
Fcc: ~/Mail/Home/INBOX
Date: Thu, 03 Mar 2011 19:43:28 -0500
Message-ID: <email address hidden>

[Martin Pool (mbp)]

rt 44398

[Andrew Bennetts (spiv)]

jubany now has bzr 2.3.1, and the imports have been retried. They are still failing, but due to “lazr.restfulclient.errors.HTTPError: HTTP Error 500: Internal Server Error” which is probably a transient timeout problem in Launchpad expected to be resolved whenever they next rollout trunk.

So probably this is now resolved, but it would be good to wait for the LP 500 error to be resolved to confirm.

[Martin Pool (mbp)]

https://bugs.launchpad.net/launchpad/+bug/716174 would make the cause of the 500 error more obvious.

Generally speaking real timeouts are 503s, but maybe not for the API.

[James Westby (james-w)]

On Fri, 25 Mar 2011 08:29:59 -0000, Andrew Bennetts <email address hidden> wrote:
> jubany now has bzr 2.3.1, and the imports have been retried. They are
> still failing, but due to “lazr.restfulclient.errors.HTTPError: HTTP
> Error 500: Internal Server Error” which is probably a transient timeout
> problem in Launchpad expected to be resolved whenever they next rollout
> trunk.

The ValueError it prints is rather odd. Do we know why that happens?

Thanks,

James

[Andrew Bennetts (spiv)]

Judging from the OOPS report it's due to OOPSes OOPS-1910A1064, which means it's probably bug 520412. I don't know why it would be affecting us now and not before, but it appears that bug is well on the way to being fixed.

[Robert Collins (lifeless)]

On Mon, Mar 28, 2011 at 12:42 PM, Andrew Bennetts
<email address hidden> wrote:
> Judging from the OOPS report it's due to OOPSes OOPS-1910A1064, which
> means it's probably bug 520412.  I don't know why it would be affecting
> us now and not before, but it appears that bug is well on the way to
> being fixed.

The fix is to just report a 400 class error to the client: the API
client is considered faulty in both cases.

[Andrew Bennetts (spiv)]

The confounding bug has been resolved, and we can see that the updated bzr has indeed fixed this problem in the package importer.

[Loïc Minier (lool)]

\o/ Thanks!