Allow building livefses against a view of the archive at a fixed point in time

[Colin Watson]

It would be useful to be able to dispatch a set of livefs builds that all reliably build from the same point in time in the archive's history, even if they're dispatched at slightly different times. We did the InRelease-by-hash work (https://bugs.debian.org/886745, and subsequent Launchpad publisher changes) as a prerequisite for this. We're now ready to consider the next stage. I believe that there are three major chunks of work here:

1) Somebody needs to work out the livecd-rootfs/live-build bits, which I think ought to be a Foundations job (I don't expect to do this part myself). It needs to pick up the inrelease-path option from the sources.list in the environment where livecd-rootfs is run and propagate that through while building the image, but make sure that the inrelease-path option *doesn't* end up in the sources.list in the final image (just like the way we build from ftpmaster.internal but don't actually leave that in the final image, since it wouldn't work).

2) In parallel with 1), we need to improve Launchpad's internal tracking of the history of archive index files. At the moment, the only feasible interface we could offer would be that the entity scheduling livefs builds would need to work out the SHA-256 hashes of all the relevant InRelease files and tell Launchpad what to use. This is cumbersome and hard to automate. Instead, it would be much better to make it work by timestamp (perhaps even simply having an option to pick whatever's current when the builds are scheduled), but to do that we need to convert ArchiveFile into a history table, adding date_created and date_superseded columns so that we can work out which version was live at any given time.

3) Once both 1) and 2) are done, we can add the relevant options to Launchpad's livefs build code to cause the correct sources.list to be dispatched.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dimitri John Ledkov (xnox)]

Is the status of this bug accurate? across xenial..focal spectrum?

[Dimitri John Ledkov (xnox)]

so apt in xenial doesn't have support for inrelease-path, which i guess is pre-requisite to build xenial images at point-in-time.

but other series have that, and can be used.

[Colin Watson (cjwatson)]

The status of this bug is accurate. Please see the linked merge proposals for more information; the current blocker is me dealing with a complex set of review feedback on https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/373971. After that, it will still remain for somebody to work out how to integrate this usefully into livecd-rootfs.

As far as I know, nobody has yet worked out how to eliminate magic-proxy entirely. What the pending Launchpad work will give you is a mechanism to query Launchpad for the state of a given metadata file in a given suite at a given point in time, which provides enough information to start the process. However, until PS5 exists (and a significant amount of development work on top of that) we won't actually be able to provide snapshot archives that are directly usable by apt, so livecd-rootfs is still going to have to do a fair amount of massaging of live-build, and somebody needs to dig into the exact details there.

[Seth Arnold (seth-arnold)]

Ayub, please do not change the status of bug reports. Thanks.

[Dimitri John Ledkov (xnox)]

@colin watson

After fighting with magicproxy & iptables issues. again. I am interested in getting magic proxy doing more-or-less things that might one day make things "nice".

I.e.
make magic proxy, talk to launchpad proxy to access authenticated archives without explicit username/password.

drop iptables nat rules, and instead try to simply export proxy for apt to use.

try to drop proxy, and instead make magic-proxy write out in-release-path stanzas for apt to use based on the passed in timestamp, and make livebuild / debootstrap use all that.

eventually "just" accept the mapping of archive/suites = hash as collected via launchpad api before dispatching the build.

Hopefully above things will make certain things easier as we go along.

Alternatively, maybe we could merge magic-proxy into the proxy that launchpad exports for the builds? I guess probably not, cause all of that is quite hackish.