Contents files for releases are sometimes empty

[Dan Watkins]

Both http://archive.ubuntu.com/ubuntu/dists/focal/Contents-amd64.gz and http://archive.ubuntu.com/ubuntu/dists/focal/Contents-i386.gz are currently empty (breaking apt-file, which is how I noticed).

< cjwatson> Odd_Bloke: That must be an LP bug - can you file it with us?

[Dan Watkins (oddbloke)]

We saw something similar for eoan at one point: https://bugs.launchpad.net/ubuntu/+source/apt-file/+bug/1848768

[Colin Watson (cjwatson)]

From #launchpad-ops on 2019-10-29:

14:09 * cjwatson peers at the Contents thing
14:09 <cjwatson> We must have got very unlucky with hitting whatever this bug is on the last run before eoan released, but it doesn't seem to be a totally isolated incident. The same thing happened to focal on 2019-10-26
16:32 <infinity> cjwatson: Are we racing somewhere we shouldn't be with Contents? I didn't look into it.
16:32 <infinity> But I guess speeding up the publisher would make that sort of thing more frequent.
16:33 <cjwatson> infinity: I think so, yes. Timestamps don't quite line up perfectly for that but maybe there's some buffering getting in the way
16:34 <cjwatson> The cp -a of ubuntu-overrides at the start of generate-contents-files was while the publisher was generating overrides, and it doesn't do the proper "write to .new, move into place" thing for those
16:34 <infinity> cjwatson: I forget how it all works, but in essence, we run a-f in Contents mode out of band, and suck the results in when publishing, yeah? Could it be that we just need an interim step of "copy the final results from a-f to the interim location the publisher will look at".
16:35 <infinity> Jinx, ish.
16:35 <cjwatson> We have such an interim step, it's just not quite stitched together exactly correctly
16:35 <infinity> Oops?
16:35 <cjwatson> Yes
16:36 <cjwatson> I'm guessing slightly but it looks about right

However, Tom fixed the thing that we thought was the cause there in https://code.launchpad.net/~twom/launchpad/+git/launchpad/+merge/374930; so we're going to need to investigate further.

[Colin Watson (cjwatson)]

The Contents files aren't currently empty, so this remains an intermittent problem.

[Colin Watson (cjwatson)]

This happened again on 2021-02-11 (possibly 2021-02-10).

[Brian Murray (brian-murray)]

Looking at some production Error Tracker retracer log files I found a couple of failures as early as 2021-02-10 18:03:59 and 2021-02-10 14:31:48.

[Brian Murray (brian-murray)]

This has happened again

Date: Sat, 20 Feb 2021 00:52:54 -0800 (PST)
From: Cron Daemon <email address hidden>
To: <email address hidden>
Subject: Cron <bdmurray@bizarro> /home/bdmurray/bin/contents-checker.py

http://archive.ubuntu.com/ubuntu/dists/hirsute-proposed/Contents-amd64.gz is empty!
http://archive.ubuntu.com/ubuntu/dists/hirsute/Contents-amd64.gz is empty!
http://ports.ubuntu.com/dists/hirsute-proposed/Contents-arm64.gz is empty!
http://ports.ubuntu.com/dists/hirsute/Contents-arm64.gz is empty! http://ports.ubuntu.com/dists/hirsute-proposed/Contents-armhf.gz is empty!
http://ports.ubuntu.com/dists/hirsute/Contents-armhf.gz is empty! http://archive.ubuntu.com/ubuntu/dists/hirsute-proposed/Contents-i386.gz is empty!
http://archive.ubuntu.com/ubuntu/dists/hirsute/Contents-i386.gz is empty! http://ports.ubuntu.com/dists/hirsute-proposed/Contents-ppc64el.gz is empty!
http://ports.ubuntu.com/dists/hirsute/Contents-ppc64el.gz is empty! http://ports.ubuntu.com/dists/hirsute-proposed/Contents-riscv64.gz is empty!
http://ports.ubuntu.com/dists/hirsute/Contents-riscv64.gz is empty! http://ports.ubuntu.com/dists/hirsute-proposed/Contents-s390x.gz is empty!
http://ports.ubuntu.com/dists/hirsute/Contents-s390x.gz is empty!

[Brian Murray (brian-murray)]

My cronjob runs every 4 hours so this happened between 20:53 and 00:52.

[Brian Murray (brian-murray)]

Date: Thu, 11 Mar 2021 08:53:03 -0800 (PST)
From: Cron Daemon <email address hidden>
To: <email address hidden>
Subject: Cron <bdmurray@bizarro> /home/bdmurray/bin/contents-checker.py

http://archive.ubuntu.com/ubuntu/dists/hirsute-proposed/Contents-amd64.gz is empty!
http://archive.ubuntu.com/ubuntu/dists/hirsute/Contents-amd64.gz is empty!

[Brian Murray (brian-murray)]

This resolved itself at most 4 hours after this check:

Date: Thu, 11 Mar 2021 20:52:52 -0800 (PST)
From: Cron Daemon <email address hidden>
To: <email address hidden>
Subject: Cron <bdmurray@bizarro> /home/bdmurray/bin/contents-checker.py

http://archive.ubuntu.com/ubuntu/dists/hirsute-proposed/Contents-amd64.gz is empty!

[Brian Murray (brian-murray)]

I received a notification today about the Contents files being empty but only for the -security and -updates pockets.

Date: Mon, 29 Mar 2021 12:53:33 -0700 (PDT)
From: Cron Daemon <email address hidden> To: <email address hidden>
Subject: Cron <bdmurray@bizarro> /home/bdmurray/bin/contents-checker.py

http://archive.ubuntu.com/ubuntu/dists/hirsute-security/Contents-amd64.gz is empty! http://archive.ubuntu.com/ubuntu/dists/hirsute-updates/Contents-amd64.gz is empty!
http://ports.ubuntu.com/dists/hirsute-security/Contents-arm64.gz is empty!
http://ports.ubuntu.com/dists/hirsute-updates/Contents-arm64.gz is empty!
http://ports.ubuntu.com/dists/hirsute-security/Contents-armhf.gz is empty!
http://ports.ubuntu.com/dists/hirsute-updates/Contents-armhf.gz is empty!
http://archive.ubuntu.com/ubuntu/dists/hirsute-security/Contents-i386.gz is empty!
http://archive.ubuntu.com/ubuntu/dists/hirsute-updates/Contents-i386.gz is empty!
http://ports.ubuntu.com/dists/hirsute-security/Contents-ppc64el.gz is empty!
http://ports.ubuntu.com/dists/hirsute-updates/Contents-ppc64el.gz is empty!
http://ports.ubuntu.com/dists/hirsute-security/Contents-riscv64.gz is empty!
http://ports.ubuntu.com/dists/hirsute-updates/Contents-riscv64.gz is empty!
http://ports.ubuntu.com/dists/hirsute-security/Contents-s390x.gz is empty!
http://ports.ubuntu.com/dists/hirsute-updates/Contents-s390x.gz is empty!

[Brian Murray (brian-murray)]

Actually they are probably supposed to be empty for hirsute-security and hirsute-updates but its weird that I just got notified about them as that implies they previously were not empty.

[Colin Watson (cjwatson)]

All those files have modification times of 2020-10-24, so if your script detected that they changed then I think it must be buggy.

[Brian Murray (brian-murray)]

It is the script and it had not taken into account the development release being frozen.