Soyuz should not allow duplicated packages in NEW/UNAPPROVED queue

[Colin Watson]

It's still possible for two uploads with the same package name and version to land in the unapproved queue; furthermore it is then possible to simultaneously accept both of them. Neither of these should be allowed.

OOPS-2018FTPMASTERPUBLISH1

[Celso Providelo (cprov)]

That's unfortunatelly right ...

As we discussed on IRC, the fact we allow dups in NEW & UNAPPROVED is annoying, but the fix will require a fix in nascentupload workflow which we are trying to keep quiet until we land ArchiveRework (because the code will need redesign for PPA anyway).

My suggestion is to workaround this situation with two fixes:

 * Materialize the state updates for each item inside the queue action batch, so the already existent lookup will find the just-modified candidate (this is a bug in previous implementation, one line fix)

 * Fix queue "fetch" action to use ID as a tie-breaker when writing dupes to disk, so foo_1.0.dsc will become foo_1.0.dsc-<ID> if it is already in disk.

Kamion, would you be satisfied (for a while) with this alternative plan ?

[Colin Watson (cjwatson)]

I'm not sure I understand your first fix. If it means "if an upload is already in the unapproved queue, then reject it", then that sounds good. If not, could you please explain further?

Please don't apply your second fix! If you munge the filenames around, then debdiff won't work properly because the filenames in the Files: sections of the .dsc files will be wrong, making 'queue fetch' largely useless. Please just make 'queue fetch' refuse to fetch an item if the files it wants to write already exist.

[Celso Providelo (cprov)]

You did, the code to refuse (name, version) duplication in ACCEPTED queue is already in place, but it doesn't work inside the same batch ("queue accept foo").

Ok, I didn't think about the filename mess with debdiff, thank you for pointing it.
The second fix will become:

 * Refuse to fetch queue items which will generate disk file overwrite (warn the user about the conflict)

Is it better ?

[Celso Providelo (cprov)]

prototype almost done in my `queue-ui`

[Celso Providelo (cprov)]

Workaround split into bug #65052, will keep this one as Confirmed/High, so it can be addressed properly soon.

[Robert Collins (lifeless)]

@Julian I think this is the earliest bug showing that the data model permits duplicates we want to avoid - possibly not the one blocking parallel uploaders, but related closely perhaps?

[Julian Edwards (julian-edwards)]

Rob, yes pretty much. See also bug 556839

[William Grant (wgrant)]

This was seen again in OOPS-2018FTPMASTERPUBLISH1, where two uploads with conflicting orig tarballs were both in NEW, and accepted within a single transaction.

[Julian Edwards (julian-edwards)]

Bumping to Critical because it causes OOPSes.

[Julian Edwards (julian-edwards)]

Any easy short-term fix would be to throw an error back to the user if we
detect simultaneous acceptance of conflicting packages when the archive admin
accepts a batch of packages.

Note that this generally only happens when they use the "queue" script tool as
the archive admin will try and accept package "foo" before they list how many
were uploaded. The UI package does not work like this, it requires you to
search for packages before you click on the packages so it would also be more
obvious that there are multiple uploads.

The queue script needs to die, in fire.

[Colin Watson (cjwatson)]

@dexterjohn2011, I'm not quite sure what you're doing, but this definitely does not belong on google-launchpad-migrator; it has nothing to do with Google Code project bugs. Reverting changes.

[Colin Watson (cjwatson)]

If we change this, we should probably first allow uploaders to reject their own uploads, so that it remains reasonably easy for them to supersede an upload that they realise is wrong.