Phased updates causing problems on official AMIs

[Jeremy Chadwick]

It has come to my attention that the official Canonical Ubuntu AMIs on AWS EC2 have "phased updates" enabled by default. How this was determined is documented here:

https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1979244

In this ticket, comments from me are with regards to ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20220609 (ami-0d70546e43a941d70), as well as 099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20220616 (ami-0c21533018816e490).

Use of phased updates introduces several problems, as the aforementioned ticket demonstrates. The official Ubuntu Wiki page on phased updates -- https://wiki.ubuntu.com/PhasedUpdates -- is an atrocity and reads more like someone's personal notebook rather than providing concrete factual details. https://people.canonical.com/~ubuntu-archive/phased-updates.html at least gives some insight to what packages are subject to this behaviour.

Anyway, here is the problem:

There is no way to "opt-out" of phased updates once they are in place. Use of APT::Get::Never-Include-Phased-Updates (a value of 1/true) has no bearing on the situation, as stated by juliank circa January 2021 here: https://discourse.ubuntu.com/t/phased-updates-in-apt-in-21-04/20345 -- quote "Note that this does not apply to fresh package installs."

In the case of the aforementioned ticket, phased updates were being applied to libssl3, which is critical; linux-headers package depends on this, for example, so it is considered a very critical "base" package.

Phased updates are effectively a form of A/B testing, and they are chosen/applied "randomly/gradually" over time, which mean that whether or not the "base AMI" was subject to them is entirely based on timing/chance. This is implied here: https://discourse.ubuntu.com/t/phased-updates-in-apt-in-21-04/20345/22

Users of official Ubuntu AMIs should not find variances like this. It wastes system administrator and developer time when encountering problems like those demonstrated in the ticket. We should find package installations to be reliable 99.9% of the time, and not subject to A/B testing that has edge case problems of this nature. Those who want to participate should opt in.

What I propose: going forward, Ubuntu AMIs should contain a new file, /etc/apt/apt.conf.d/25-disable-phased-updates, which contains the following line:

APT::Get::Never-Include-Phased-Updates true;

Whatever scripts/etc. are used for building the base AMIs may want to put this into place directly, or alternately, may have to use -o APT::Get::Never-Include-Phased-Updates=true (or whatever the correct equivalent is) when using apt/apt-get/etc. to ensure this doesn't happen.