import fails when lp branch has been push --overwrite'n

[Vincent Ladeuil]

The package importer is tricked when packagers do a push --overwrite.

To fix the importer, one should:

  requeue_package.py --full <package>

The traceback signature is:

  AssertionError:<module>:main:find_unimported_versions:check

[Vincent Ladeuil (vila)]

Doing 'requeue_package.py --full <package>' isn't enough, more investigation needed.

[Max Bowsher (maxb)]

The armel-cross-toolchain-base instance of this failure is suitable for requeue_package.py --full

[Max Bowsher (maxb)]

requeue_package.py --full bootchart

[Max Bowsher (maxb)]

bridge-utils is an interesting one. lucid maverick and natty are all on the same version in the archive. This version was committed in the branch by Steve Langasek, and is tagged. In the natty branch only, the importer has added an additional revision adding some .gitignore files (presumably present in the archive but not the branch. But, this extra revision is only present in the natty branch. Hmm.

[James Westby (james-w)]

On Fri, 10 Jun 2011 19:54:41 -0000, Max Bowsher <email address hidden> wrote:
> The armel-cross-toolchain-base instance of this failure is suitable for
> requeue_package.py --full

Done.

Thanks,

James

[Max Bowsher (maxb)]

casper: karmic and earlier are importer-based branches lucid and later are direct packager maintained, with history going back to Arch branches.

cloud-init: Now direct packager maintained branch

[Max Bowsher (maxb)]

desktopcouch: Some fairly mad stuff going on here, with 0.6.6 being committed after 0.6.9b. Some direct commits by packagers. All a bit messy really.

dpkg: There's a 1.16.0~ubuntu8 in the natty branch, and there's a 1.16.0~ubuntu8 in the oneiric branch, and they're different packages!?!! :-(

[Max Bowsher (maxb)]

ecl: Direct packager maintained branch

euca2ools: Direct packager maintained branch

[James Westby (james-w)]

On Fri, 10 Jun 2011 20:15:51 -0000, Max Bowsher <email address hidden> wrote:
> requeue_package.py --full bootchart

Done as well.

Would you like the ones that you state are "direct packager maintained"
branches requeued as well?

Thanks,

James

[Max Bowsher (maxb)]

On 15/06/11 20:29, James Westby wrote:
> Would you like the ones that you state are "direct packager maintained"
> branches requeued as well?

Not right now - I said that for ones which were apparent had substantial
non-importer history, but which I had otherwise not examined closely.

I intend to revisit them later (but for now I've stopped working on
failures until bug 797088 is fixed, as it prevents timely feedback on
whether frobbed imports actually start working).

Max.

[Max Bowsher (maxb)]

Please requeue --full ubuntu-defaults-builder, the import branch for which has just been --overwritten with human-managed history

[Max Bowsher (maxb)]

> Please requeue --full ubuntu-defaults-builder

Done with vila on IRC

[Martin Pool (mbp)]

this is also currently affecting dpkg, which cjwatson would like fixed, but it seems like a requeue --full would be disruptive because that would orphan existing branches based off the import?

How should we fix this? Essentially do a conceptual pull --overwrite into the importer, telling it to treat the lp revisions as definitive, and to forget its own records?

[James Westby (james-w)]

On Thu, 13 Oct 2011 10:02:28 -0000, Martin Pool <email address hidden> wrote:
> this is also currently affecting dpkg, which cjwatson would like fixed,
> but it seems like a requeue --full would be disruptive because that
> would orphan existing branches based off the import?

I don't see that the import has failed, so I'm not sure what the issue
with dpkg is.

> How should we fix this? Essentially do a conceptual pull --overwrite
> into the importer, telling it to treat the lp revisions as definitive,
> and to forget its own records?

vila added a script or an option to a script to have it forget whats in
its db without reimporting the revisions I think.

Thanks,

James

[Martin Pool (mbp)]

On 14 October 2011 09:17, James Westby <email address hidden> wrote:
> On Thu, 13 Oct 2011 10:02:28 -0000, Martin Pool <email address hidden> wrote:
>> this is also currently affecting dpkg, which cjwatson would like fixed,
>> but it seems like a requeue --full would be disruptive because that
>> would orphan existing branches based off the import?
>
> I don't see that the import has failed, so I'm not sure what the issue
> with dpkg is.

It seems to have passed now, I need to read the scrollback to work out
what was done about it.

>
>> How should we fix this?  Essentially do a conceptual pull --overwrite
>> into the importer, telling it to treat the lp revisions as definitive,
>> and to forget its own records?
>
> vila added a script or an option to a script to have it forget whats in
> its db without reimporting the revisions I think.

Perhaps we can document that or smooth it off.

m

[Steve Langasek (vorlon)]

I think Martin's suggestion for a "conceptual pull --overwrite" to treat the lp versions as authoritative would be the correct course of action here.

And I would appreciate this sooner rather than later, since I've been guilty of a couple of these push --overwrites to LP myself (probably including the dpkg one), which means the affected packages include a disproportionate number of packages that I'd like to be using via UDD. ;)

[Steve Langasek (vorlon)]

Are there any prospects of progress on this in the near term?

If not, could someone force the importer to regard the existing branch revisions as authoritative for the following packages?:

  debhelper
  sysvinit
  qemu-linaro
  plymouth
  u-boot-linaro

[James Westby (james-w)]

On Thu, 26 Jan 2012 00:48:14 -0000, Steve Langasek <email address hidden> wrote:
> Are there any prospects of progress on this in the near term?
>
> If not, could someone force the importer to regard the existing branch
> revisions as authoritative for the following packages?:
>
> debhelper
> sysvinit
> qemu-linaro
> plymouth
> u-boot-linaro

Hi Steve,

Apologies for the delay on this. I've just poked each of these
packages. Let me know if any of them don't catch up (should be done in
an hour or two at most)

Thanks,

James

[Max Bowsher (maxb)]

The importer made a horrid mess of plymouth which I've just unpicked, and then done manual SQL on meta.db with the aim of convincing the importer to accept the existing revisions as ok.

[James Page (james-page)]

Please could the tomcat6 branch have magic done to it as well as its suffering from this issue.

Thanks

James

[Max Bowsher (maxb)]

tomcat6 tags fixed up and import in progress

[Dimitri John Ledkov (xnox)]

I would really love the magic to be done to initramfs-tools please.

Thank you in advance,
Dmitrijs.

[Colin Watson (cjwatson)]

Also grub-gfxpayload-lists.

[Logan Rosen (logan)]

Can somebody perform this magic on unixcw as well, please? I'd love to be able to update the version through bzr.

[Dimitri John Ledkov (xnox)]

Request to requeue gedit, see duplicate bug.

[Thomas Bechtold (toabctl)]

Also package "icu" is affected.

[Darik Horn (dajhorn)]

Package mountall is affected. Version 2.36.3 is currently published in precise-updates but `bzr branch lp:ubuntu/precise/mountall` returns version 2.36.

[Andrew Starr-Bochicchio (andrewsomething)]

Could someone please requeue bzr itself? I'd love to actually use UDD to merge bzr!

[Barry Warsaw (barry)]

Can we please prevent all non-importer pushes to UDD branches? In the normal (i.e. not failing) course of events, IME it's always better to let the importer do it rather than push your own local branch, --overwrite or not.

[Tais P. Hansen (taisph)]

Could someone please requeue golang?

[Steve Langasek (vorlon)]

On Thu, Feb 27, 2014 at 10:20:09PM -0000, Tais Plougmann Hansen wrote:
> Could someone please requeue golang?

Done.

[Dimitri John Ledkov (xnox)]

Steve, how do you requeue branches? cause the way i was doing it, was wiping out history. Go into the database and update the expected revid & revhash? It would be nice to e.g. document it here.

[Steve Langasek (vorlon)]

On Fri, May 02, 2014 at 02:48:52PM -0000, Dimitri John Ledkov wrote:
> Steve, how do you requeue branches? cause the way i was doing it, was
> wiping out history. Go into the database and update the expected revid &
> revhash? It would be nice to e.g. document it here.

In general, a simple requeue of 'scripts/bin/requeue-package <packagename>'.
For the various packages that seem to have been forcibly requeued, the tags
on the branches have been moved but the database has not been updated; so
simply deleting the revid from the database and requeuing is sufficient to
get the database synced with the branch. (It's certainly not worth trying
to undo the changes on the branch, IMHO.)

[Steve Langasek (vorlon)]

Correction, it seems that what's happened in most cases is that thedatabase has been updated but the new tags were never pushed, so the database is out of sync with the branch. Deleting the updated database records and requeuing is enough in most cases toget the db restored to its original pre-clobber state.

[William Grant (wgrant)]

On 13/05/14 15:49, Steve Langasek wrote:
> On Fri, May 02, 2014 at 02:48:52PM -0000, Dimitri John Ledkov wrote:
>> Steve, how do you requeue branches? cause the way i was doing it, was
>> wiping out history. Go into the database and update the expected revid &
>> revhash? It would be nice to e.g. document it here.
>
> In general, a simple requeue of 'scripts/bin/requeue-package <packagename>'.
> For the various packages that seem to have been forcibly requeued, the tags
> on the branches have been moved but the database has not been updated; so
> simply deleting the revid from the database and requeuing is sufficient to
> get the database synced with the branch. (It's certainly not worth trying
> to undo the changes on the branch, IMHO.)

I investigated this some time ago, and I ended up with the understanding
that a requeue-package --full was always destined to work once but then
subsequently fail, as it updated the tags in the database but didn't
overwrite the tags in Launchpad's copy of the branch. I possibly
misunderstand or misremember.

[Jean-Philippe Orsini (jfi)]

Could someone requeue psensor please? (see http://package-import.ubuntu.com/status/psensor.html#2014-04-08%2014:18:53.940182)

Thanks!

[Matthew Krafczyk (krafczyk-matthew)]

could someone requeue openafs please? (http://package-import.ubuntu.com/status/openafs.html) reports:

This page was last updated at 2014-11-13T10:15:01.695936 UTC.

Failed at 2013-05-14 02:49:02.774379

https://bugs.launchpad.net/udd/+bug/714622 The importer's local state database regarding tags that it has previously imported disagrees with the actual tags in the branch. Something has been changed, and further imports have been ceased pending manual review.

Traceback (most recent call last):
  File "/srv/package-import.canonical.com/new/scripts/bin/import-package", line 5, in <module>
    sys.exit(main(sys.argv))
  File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py", line 1178, in main
    only_before=options.only_before)
  File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py", line 1060, in _import_package
    revid_db, bstore, possible_transports=possible_transports)
  File "/srv/package-import.canonical.com/new/scripts/udd/scripts/import_package.py", line 696, in find_unimported_versions
    has_version = revid_db.check(importp.version, revid, suite, db.branch)
  File "/srv/package-import.canonical.com/new/scripts/udd/icommon.py", line 1170, in check
    self.package, unicode(version), suite))
AssertionError: ('<email address hidden>', 'd944a861afc9374c256adf327bcbe9c4df20b184') != (<email address hidden>', u'791311cd7ceea60cbb9af0a675d04b6fe70d8a4b') for openafs 1.4.11+dfsg-1 in karmic, something has changed