duplicate orig for "linux" package in hardy

[Kees Cook]

Today, I encountered the following error from Soyuz when attempting to unembargo the Hardy linux package:

linux 2.6.24-28.80 in hardy (linux_2.6.24.orig.tar.gz already exists in destination archive with different contents.)

However, both the PPA and the archive contained the correct hash (e4aad2f8c445505cbbfa92864f5941ab). With wgrant and mbarnett's help the problem hash was found here:

https://edge.launchpad.net/ubuntu/+archive/primary/+sourcepub/277962/+listing-archive-extra

It seems the code doing the comparison was sensitive to database sort order, which may have changed during the recent postgresql upgrades.

It seems like this situation needs to be detected for the rest of the package database, and that the orig-hash-checker code probably needs to be updated to deal with this broken state more gracefully.

[William Grant (wgrant)]

The issue is that pub 277962's linux_2.6.24.orig.tar.gz was somehow replaced with a different one. Archive.getFilesAndSha1s assumes that there is only one hash for a particular file in an archive, and is sensitive to DB ordering when there are multiple.

The best fix is probably to identify and expire all the conflicting files, since none of them should be published any more.

[Julian Edwards (julian-edwards)]

We don't expire old files until the series is obsolete.

Thanks for the analysis William.

[William Grant (wgrant)]

We don't normally, no. But these are broken, so why not do it manually and unbreak the DB?

[Julian Edwards (julian-edwards)]

So looking at this a bit more, the assumption that "there is only one hash for a particular file in an archive" is correct - that should always be the case, so what Archive.getFilesAndSha1s() and the package copier is doing is also correct. Sort order has no bearing on this at all.

The problem is that there is a conflicting file *already in the archive*. How that got there is what I am most interested in now.

Kees, do you have any idea how we'd end up with two different linux_2.6.24.orig.tar.gz files?

lpmain_staging=> select distinct LibraryFileContent.sha1 from LibraryFileAlias, LibraryFileContent, SourcePackagePublishingHistory, SourcePackageReleaseFile, SourcePackageRelease
where SourcePackagePublishingHistory.archive = 1and SourcePackageRelease.id = SourcePackagePublishingHistory.sourcepackagerelease
and SourcePackageReleaseFile.sourcepackagerelease = SourcePackageRelease.id
and LibraryFileAlias.id = SourcePackageReleaseFile.libraryfile
and LibraryFileAlias.filename = 'linux_2.6.24.orig.tar.gz'
and LibraryFileContent.id = LibraryFileAlias.content;
                   sha1
------------------------------------------
 b7b63f52551f9e4bf2e5ba902f8385fbefc5d80c
 ccccdc4759fd780a028000a1b7b15dbd9c60363b
(2 rows)

[Julian Edwards (julian-edwards)]

The md5s might be more useful since that's what's in the UI:

               md5
----------------------------------
 e4aad2f8c445505cbbfa92864f5941ab
 f09806748f6809d5f7d2a1859240e1a5

[Julian Edwards (julian-edwards)]

The one that you say is broken is this source:
https://edge.launchpad.net/ubuntu/+source/linux/2.6.24-5.9

[Jamie Strandboge (jdstrand)]

This happened again today when trying to publish a security update for hardy-security. I tried both with our unembargo script and on cocoplum:

$ LPCONFIG=production /srv/launchpad.net/codelines/soyuz-production/scripts/ftpmaster-tools/unembargo-package.py -p ubuntu-security -d ubuntu -s hardy-security linux
...
2010-11-29 23:24:13 ERROR linux 2.6.24-28.81 in hardy (linux_2.6.24.orig.tar.gz already exists in destination archive with different contents.)
Confirm this transaction? [yes, no] no

This is preventing publication of an important kernel security update.

[Jamie Strandboge (jdstrand)]

Ok, spm cowboyed a very temporary fix for me for the kernel publication (and quickly removed it). It would be nice if this could get resolved before the next kernel update.

[William Grant (wgrant)]

http://pastebin.ubuntu.com/516480/ is the patch we've used for the last two incidents.

We can either fix the data or fix the hash check to only respect the latest hash for each filename.

[Julian Edwards (julian-edwards)]

Jamie, any answer to my question in comment #4?

[Julian Edwards (julian-edwards)]

On Tuesday 30 November 2010 10:02:59 you wrote:
> Jamie, any answer to my question in comment #4?

Ah wgrant just reminded me that we are accepting duplicate files after the old
file is deleted. That should not happen.

[Jamie Strandboge (jdstrand)]

Julian,

I don't know how 2.6.24-5.9 got a different orig.tar.gz. That seems to have happened some time in the Hardy development cycle (ie, outside of any security team updates) since the release kernel for Hardy is 2.6.24-16.30 and iirc gutsy had 2.6.22. https://launchpad.net/ubuntu/+source/linux/+publishinghistory is giving me timeout errors so I can't really do more than guess atm.

[Julian Edwards (julian-edwards)]

Thanks Jamie. Don't worry about it for now, I'm landing a change that will
prevent anyone from doing it again.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r12009 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12009>.