multiple copies of orig.tar.gz's in the librarian

[James Troup]

The sync-source tool deliberately explodes when it finds more than one orig.tar.gz in the archive for a given source_version. This appears to be the case for, e.g. advi_1.6.0.orig.tar.gz. I downloaded both copies, and confirmed that the md5sums and sizes match. Even so, there should only be one copy in the librarian.

*****
<SourcePackageFilePublishing at 0x2aaab5737f50>
filename: advi_1.6.0.orig.tar.gz
alias: 1268214
distrorelease: hoary, component: universe, source: advi, status: 2
*****
<SourcePackageFilePublishing at 0x2aaab5737fd0>
filename: advi_1.6.0.orig.tar.gz
alias: 1331574
distrorelease: breezy, component: universe, source: advi, status: 2
*****
<SourcePackageFilePublishing at 0x2aaab71b30d0>
filename: advi_1.6.0.orig.tar.gz
alias: 1331574
distrorelease: dapper, component: universe, source: advi, status: 3
*****
<SourcePackageFilePublishing at 0x2aaab71b3150>
filename: advi_1.6.0.orig.tar.gz
alias: 1331574
distrorelease: dapper, component: universe, source: advi, status: 2
E: advi_1.6.0.orig.tar.gz (from advi) returns multiple IDs for orig.tar.gz. Help?

[Andrew Bennetts (spiv)]

Not directly helpful, but this background information about how the Librarian handles duplicates may be of interest:

The librarian tries to ensure that identical files are only stored once (and so the LibraryFileContent table will only have one row for that file), but by design allows duplicate aliases to that content. (Additionally, there can be duplicate content if the same new file is uploaded simultaneously in two seperate but concurrent transactions, but the Librarian GC process will find and collapse duplicate content rows daily).

Basically, this means this constraint needs to be enforced somewhere other than the librarian. A workaround might be to try relying on the librarian's existing duplicate detection, i.e. check if the duplicate aliases are linked to the same content or not, but I don't think this will be bulletproof.

Judging from a bit of grepping, the place where these files are added is nascentupload.py, in insert_source_into_db, so this would be the obvious place to start fixing.

[Celso Providelo (cprov)]

RF 3476, but we still missing a magic SQL to fix the duplicated files in the production DB

[Matt Zimmerman (mdz)]

This is blocking certain pending archive requests, such as bug #41213, so raising severity.

Celso, it looks like you're working on this, so I'm assigning to you; hope that's OK.

[Daniel Silverstone (dsilvers)]

I have been working on the SQL to fix this up.

I need stuart to finish a librariangc run before I can continue

[Stuart Bishop (stub)]

librarian-gc.py run has completed.

[Celso Providelo (cprov)]

Daniel, Did you check the results ? should we continue the dapper-autotest process and see what happen ?

[Scott James Remnant (Canonical) (canonical-scott)]

Still seeing this today.

[Adam Conrad (adconrad)]

And 4 days after the last comment, I just ran across this with a sync I was attempting, and it's blocking some bugfixes we need to grab from Debian. If there's anything I can do to help get this tested and rolled out, let me know.

[Matt Zimmerman (mdz)]

I've opened an Ubuntu task to track the process of cleaning up after this bug, since the bug itself has been fixed

[Matt Zimmerman (mdz)]

Here's my analysis of the duplicates that kiko brought to my attention, which were found with a database query.

Orig needs fixing in pool:
 digikam_0.8.1.orig.tar.gz | 2
 kdissert_1.0.5.debian.orig.tar.gz | 2

Duplicate orig, but pool has the correct one:
 gossip_0.10.2.orig.tar.gz | 2
 xfce4-mixer_4.3.0svn+r19775.orig.tar.gz | 2

Needs upload:
 at_3.1.8-11ubuntu5.dsc | 2
 at_3.1.8-11ubuntu5.tar.gz | 2
 ebtables_2.0.6-3ubuntu1.diff.gz | 2
 ebtables_2.0.6-3ubuntu1.dsc | 2
 xfce4-dev-tools_4.3.3svn-r20589-0ubuntu1.diff.gz | 2
 xfce4-dev-tools_4.3.3svn-r20589-0ubuntu1.dsc | 2

Not currently published:
 gazpacho_0.6.2-0ubuntu4.diff.gz | 2
 gazpacho_0.6.2-0ubuntu4.dsc | 2
 gok_1.0.5-1ubuntu5.diff.gz | 2
 gok_1.0.5-1ubuntu5.dsc | 2
 gtksourceview_1.6.1-0ubuntu2.diff.gz | 2
 gtksourceview_1.6.1-0ubuntu2.dsc | 2
 hal_0.5.7-1ubuntu9.diff.gz | 2
 hal_0.5.7-1ubuntu9.dsc | 2
 hotkey-setup_0.1-15build1.dsc | 2
 hotkey-setup_0.1-15build1.tar.gz | 2
 initramfs-tools_0.40ubuntu26.dsc | 2
 initramfs-tools_0.40ubuntu26.tar.gz | 2
 kmplayer_0.9.1.99+0.9.2-pre3-0ubuntu1.diff.gz | 2
 kmplayer_0.9.1.99+0.9.2-pre3-0ubuntu1.dsc | 2
 libnjb_2.2.4-3.diff.gz | 2
 libnjb_2.2.4-3.dsc | 2
 ubuntu-meta_0.110.dsc | 2
 ubuntu-meta_0.110.tar.gz | 2
 udev_079-0ubuntu9.diff.gz | 2
 udev_079-0ubuntu9.dsc | 2

I went ahead and made no-change uploads for the three packages which could be easily fixed that way

[Celso Providelo (cprov)]

Sync Tool should support multiple LFA (Library File Alias) which point to the same LFC (Library File Content), since, in fact, they are the same file; and only aborts the procedure when the contents diverge.

The agglomeration of the multiple LFA must be done at some point, but as far I can tell this condition is harmless to the system, since the download result will be the same.

This patch might not apply clearly to the production version of the sync-tool, since it appears to have local changes.

[James Troup (elmo)]

Celso Providelo <email address hidden> writes:

> ** Attachment added: "Proposed Sync Tool Fix"
> http://librarian.launchpad.net/2538918/fix_sync_tool_to_support_multiple_LFA_with_same_LFC.diff

Err, the sync tool already does this.

--
James

[Celso Providelo (cprov)]

As discussed with kiko, style fix and plan for redesign the entire script.

[Celso Providelo (cprov)]

James, where is this code and why does it still failing like reported in bug #41487 ?

[Andrew Bennetts (spiv)]

Celso, is it possible to construct a test case for this so that you can reproduce it, and be sure that the proposed fix actually works? There appears to be a fair bit of guesswork going on, it would be nice to be able to reproduce the problem in a test environment.

Also, having tests in your patch will help it pass code review...

[Celso Providelo (cprov)]

Andrew Bennetts wrote:
> Celso, is it possible to construct a test case for this so that you
> can reproduce it, and be sure that the proposed fix actually works?
> There appears to be a fair bit of guesswork going on, it would be
> nice to be able to reproduce the problem in a test environment.

If by this, you mean the problem itself (as we should not accept
duplicated filename with different content), yes, we already have it
integrated in the doc/zzz-soyuz-set-of-uploads.txt, so, we are sure we
are not making the same mistake again.

However, this last fix is in sync-tool, to deal with the corrupted
content (even if I think the multi LFAs to a single LFC isn't a real
corruption, since the system does support it). And also a fix to apply
experientially in a portion of code that isn't even kept up to date in
LP tree :( , no standards, no patterns, no test, it's a contributed code
atm.

I appreciate your concern and agree with it, I have plan to redesign
each single tool located in scripts/ftpmaster/ (as discussed some time
ago in lp-reviews with bjornt) and it include support classes for each
script, functional and doc tests, standard command-line options, etc.

> Also, having tests in your patch will help it pass code review...

I count with your understanding in this situation, it can't change
magically, redesign in this land would require time and more hands to
not break the dapper release process.

[]
--
Celso Providelo <email address hidden>
Canonical Ltd - http://www.canonical.com

[Matt Zimmerman (mdz)]

The situation turned out to be (possibly) somewhat better than I originally thought; it was only digikam where the .orig was mismatched relative to Debian. In the kdissert case, the Ubuntu .dsc was wrong, but the .orig in the pool matched Debian.

So I've uploaded digikam with a fresh renamed .orig, and kdissert with only a new revision and the same orig, which should get the pool into a consistent state.

Can someone confirm what the database says, so we know if kdissert needs the same treatment or not?

[Celso Providelo (cprov)]

yes, kdissert needs new upload with fresh orig version to overlap the differences between the archive and the DB properly, as you did for digikam.

After some investigation with kiko we figured out that what was in archive as Kdissert_1.0.5.debian.orig.tar.gz, didn't fit with what the systems recognizes as such.

What exactly happened is a new upload was accepted with a new orig content but the same name, but at the end, the publisher refused to overwrite the file in archive. (boom, inconsistency between archive and model, but not even a single warn).

The DB has, let's say for comparison sake, kdissert_1.0.5.debian2.orig.tar.gz contents but named as the former.

We expect a new full upload would fix the archive snapshot, but the history will be broken.

[Matt Zimmerman (mdz)]

On Thu, May 11, 2006 at 08:49:58PM -0000, Celso Providelo wrote:
> yes, kdissert needs new upload with fresh orig version to overlap the
> differences between the archive and the DB properly, as you did for
> digikam.

I uploaded it earlier today; should be fine now.

--
 - mdz

[Celso Providelo (cprov)]

Matt Zimmerman wrote:
> On Thu, May 11, 2006 at 08:49:58PM -0000, Celso Providelo wrote:
>> yes, kdissert needs new upload with fresh orig version to overlap the
>> differences between the archive and the DB properly, as you did for
>> digikam.
>
> I uploaded it earlier today; should be fine now.
>

Yes, both (digikam & kdissert) seem fine now from a.u.c

thank you

--
Celso Providelo <email address hidden>
Canonical Ltd - http://www.canonical.com

[Celso Providelo (cprov)]

code is fixed and current versions are ok.