custom uploads can be published in a later publisher cycle from the rest of the upload

[Andy Whitcroft]

When we are signing kernels we copy the kernel triplet to the signing PPA as a set (linux, linux-signed, and linux-meta). linux-signed is deliberatly copies without binaries so it will pick up the correctly signed content. As this is a private-ppa the linux-signed builds do not start until the next publisher run completes and publishes all of the sources and binaries for linux and linux-meta. These builds regularly fail with the dists component from the custom upload apparently missing. These are then found to be in place and a retry will succeed.

cjwatson looked into the history of a sample event:

10:04:24 cjwatson | linux-signed-fips copy job was processed at 17:34:32
10:07:02 cjwatson | The source was published starting at 17:42:18
10:07:33 cjwatson | The publish-distro run that that publication was part of started at 17:41:05
10:08:15 cjwatson | However, the preceding process-accepted run finished at 17:28:29, and the importance of that will become clear in a moment
10:08:38 cjwatson | Because custom uploads aren't modelled as publication history records, we instead have to send copies of them through the upload queue
10:10:38 cjwatson | That means that the linux-signed-fips copy job turned into (a) a copy of the source which was published in the next relevant publish-distro run, and (b) a queue entry that was
                  | processed in the next relevant process-accepted run and then published in the next relevant publish-distro run after that
10:11:44 cjwatson | In this case the next relevant process-accepted run started at 17:45:06 and processed the linux-fips copy at 17:45:30
10:12:10 cjwatson | And the actual publication of that finished at 17:56:40
10:13:40 cjwatson | So your case fails if the copy is processed between process-accepted --ppa and publish-distro --private-ppa in a single cron.publish-ppa cycle, which is going to be pretty common since
                  | that covers quite a lot of time
10:14:07 cjwatson | We could perhaps mitigate this by splitting process-accepted --ppa up into --ppa and --private-ppa the way we do for publish-distro, and then the window would be narrower, but it
                  | wouldn't close it entirely
10:14:28 cjwatson | The only way to close it entirely would be to refactor how custom uploads work
10:14:47 apw | the dists part is only in place on the next publisher cycle for that ppa ?
10:14:50 cjwatson | Right

[Colin Watson (cjwatson)]

This can fail for copies to public PPAs too, although the window is narrower in that case.

[Colin Watson (cjwatson)]

Dispatching private builds immediately (https://code.launchpad.net/~cjwatson/launchpad/build-private-bpb-immediately/+merge/345104 and its prerequisite) might help in this case, although only by chance. While the linux-signed build would initially be dispatched earlier, it would immediately dep-wait because the main linux packages wouldn't have been published yet. buildd-retry-depwait would retry it once the linux packages had been published, but that currently runs hourly, so it would have a reasonable chance of ending up in a good place.

This wouldn't be a proper fix, of course, but I think it would reduce the practical impact.

[Colin Watson (cjwatson)]

We now dispatch private builds immediately, reducing the practical impact of this bug as described in my previous comment.