ubuntutools.archive.UbuntuSourcePackage().pull() fails (take 2)

[Robie Basak]

Steps to reproduce:

import ubuntutools.archive
ubuntutools.archive.UbuntuSourcePackage('apache2', '2.2.8-1ubuntu0.4').pull()

This raises the following exception:

DownloadError('https://launchpad.net/ubuntu/+archive/primary/+sourcefiles/apache2/2.2.8-1ubuntu0.4/apache2_2.2.8-1ubuntu0.4.dsc: 500 Internal Server Error',)

On Focal, with python3-ubuntutools 0.176.

See also: bug 1860456. The logic was adjusted there, so this might be fallout from that.

[Dan Streetman (ddstreet)]

@cjwatson apologies, I added you before I noticed @rbasak changed the target from ubuntu-dev-tools to launchpad itself (and then removed you after I noticed, since you already get bug email for launchpad itself).

[William Grant (wgrant)]

Violations of the source version uniqueness rule in the Ubuntu primary archive (apport is certainly creative):

 archive | name | version | array_agg
---------+-----------------------------+----------------------------+--------------------------------------------------------------------------------------------------------------------------
       1 | apache2 | 2.2.8-1ubuntu0.4 | {365510,370076}
       1 | apport | 0.36 | {135051,135052,135053,135054,135055,135056,135057,135058,135059,135060,135061,135062,135063,135064,135065,135066,135067}
       1 | compiz-plugins | 0.5-0ubuntu4 | {121017,121018}
       1 | cowbell | 0.2.6.2-0ubuntu1 | {94928,94932}
       1 | ebtables | 2.0.6-3ubuntu1 | {96416,96418}
       1 | eog | 2.14.2-0ubuntu1 | {103582,103600}
       1 | ffmpeg | 3:0.cvs20070307-5ubuntu7.2 | {373961,291349}
       1 | gtksourceview | 1.6.1-0ubuntu2 | {97583,97586}
       1 | guml | 0.3 | {82734,107244}
       1 | hal | 0.5.7-1ubuntu9 | {96078,96079}
       1 | hotkey-setup | 0.1-15build1 | {96014,96015}
       1 | hplip | 2.7.7.dfsg.1-0ubuntu5.1 | {328647,175617}
       1 | htcheck | 1:1.2.3-2ubuntu1 | {100880,100881}
       1 | initramfs-tools | 0.40ubuntu26 | {95928,95929}
       1 | jokosher | 0.1-0ubuntu1 | {112286,112122}
       1 | knapster2 | 0.5-7 | {111412,111489}
       1 | ktorrent | 2.0.2-0ubuntu2 | {121035,121046}
       1 | language-pack-gnome-qu-base | 1:6.10+20060924 | {120109,119750}
       1 | language-pack-kab-base | 1:6.10+20060924 | {120161,119802}
       1 | language-pack-kde-kab-base | 1:6.10+20060924 | {120208,119849}
       1 | language-pack-kde-tlh-base | 1:6.10+20060924 | {119899,120258}
       1 | language-pack-qu-base | 1:6.10+20060924 | {120305,119946}
       1 | language-selector | 0.1.26 | {121062,121049}
       1 | libapache-authznetldap-perl | 0.07-4 | {323988,77504}
       1 | libnjb | 2.2.4-3 | {95919,95918}
       1 | libwnck | 2.14.2-0ubuntu1 | {103584,103588}
       1 | murasaki | 0.8.11-3 | {84935,107431}
       1 | penguintv | 1.02-0ubuntu3 | {100549,100548}
       1 | premail | 0.46-9 | {65744,262421}
       1 | telepathy-feed | 0.11-0ubuntu1 | {112081,112082}
       1 | ubuntu-meta ...

[William Grant (wgrant)]

There are more violations in PPAs (Canonical-internal link): https://pastebin.canonical.com/p/MtRX9MbNdD/

[Robie Basak (racb)]

Thank you for investigating. I was under the impression that a database constraint prevented this. I guess not!

I am making the assumption in git-ubuntu that the source of a particular (distribution, package name, package version) is unique. Does this finding violate that, or are the underlying sources still the same?

[Colin Watson (cjwatson)]

The invariant that you're assuming is a reasonable one, but unfortunately there are some anomalies due to bugs (hopefully mostly historical; the newest instance I can find is from January 2018). If possible, our intention is to clean up production data and then add a constraint so that more instances can't arise in future.

[Robie Basak (racb)]

Thanks - that will work well for me. As I'm in the process of reimporting everything already, and those reimports are failing when the invariant is found to not hold, I think that means that we won't be able to get data into git-ubuntu that will later not reflect Launchpad's data.

[Colin Watson (cjwatson)]

All the primary archive cases are now fixed. I don't think it's worth proactively working through the cases in PPAs, but we can look into these on a case-by-case basis if needed.