Unexpected / unwanted unattended-upgrades behaviour after kernel upgrade when Livepatch enabled

[mev]

Following the resolution for bug #1747499, after a kernel upgrade when Livepatch is enabled, the current behaviour in unattended-upgrades (2.3ubuntu0.2 and later) is not to touch /var/run/reboot-required so as not to confuse users with two separate messages calling for a restart in motd. This functionality is implemented in the script at /etc/kernel/postinst.d/unattended-upgrades.

While this works as intended in terms of suppressing an extra message in motd, it defeats the ability of unattended-upgrades to restart automatically with the new kernel, which is reliant on /var/run/reboot-required being present.

This is unexpected / unwanted behaviour in scenarios where a) Livepatch is being used to provide fast-response kernel patching; and b) Unattended-Upgrade::Automatic-Reboot is set to true, to enable automatic reboots during a regular maintenance window. In this case, without administrative intervention, the system could never boot into the new kernel even though it would be expected to, leaving Livepatch to do all the heavy lifting indefinitely, and unnecessarily.

I believe this counts as a regression caused by the resolution to bug #1747499. It also has the potential to be a security threat if Livepatch doesn't work comprehensively for a particular kernel flaw, and an administrator is reliant on expected behaviour according to unattended-upgrades settings.

Potential options for a fix that come to mind:
1. Revert to original behaviour in /etc/kernel/postinst.d/unattended-upgrades, and change the ***System restart required*** message to be less alarming or confusing when the cause is a kernel upgrade that's being patched by Livepatch.
2. Add an extra configuration setting (eg Unattended-Upgrade::Automatic-Reboot-After-Livepatch) that triggers a reboot when it's 'recommended' by Livepatch, not reliant on the presence of /var/run/reboot-required.
3. Add support in /etc/kernel/postinst.d/unattended-upgrades for an extra file somewhere. When present, /var/run/reboot-required is always touched, even if Livepatch is enabled.

(This is my first time reporting a bug in this system, and I apologise if I haven't followed the usual descriptive format.)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Julian Andres Klode (juliank)]

How updates and optional reboot requests are surfaced is a pretty complex, I do not feel we can adequately address this in a bug report, it's part of a much wider story about the Pro client.

I'll go ahead and assign this to server as they own the pro client integration bits and are in the process of unifying that all, because I think they need to think about this for a bit.

[mev (mev1)]

If I may: I don't think this is a hugely technical issue. It comes down to unintended consequences caused by the fix for bug #1747499 - while its purpose was to resolve confusing user messaging, it probably took the wrong approach.

A straightforward fix would be to:
1. Revert the changes from bug #1747499.
2. Change the "System restart required" text applied by /usr/share/update-notifier/notify-reboot-required to something more specific and user-friendly, like "System restart required to finish applying updates."
3. Change the output from canonical-livepatch kernel-upgrade-required so that it doesn't reference restarting the system. (As we've seen this month - July 2023 - the message isn't always valid anyway.) For example the text could be simply: "Livepatch has fixed kernel vulnerabilities. Kernel upgrade recommended when available."

I believe that's all that is required, although I do understand that it's complicated somewhat by the functionality being split across different packages with different people responsible, and that it involves internationalised strings.

[Robie Basak (racb)]

Thanks mev.

I wonder if there's a separate, deeper question here rather than just the messaging. How does one configure the system so that it automatically reboots when required, except when livepatch is sufficient? If that's not what you were raising, then perhaps we should have a separate bug to track that question.

[mev (mev1)]

Hi Robie -

I think that's a separate issue!

As far as I know, there's no way of measuring the sufficiency of Livepatch. A new kernel will usually have fixes for other bugs and less-impactful vulnerabilities on top of what Livepatch is taking care of, and it's impossible to know whether those fixes will be of significance to the user.

If a user wants to rely on Livepatch entirely (or 'until further notice'), unattended-upgrades can be set to blacklist automatic kernel upgrades.

I certainly don't see any value, in any circumstance, of installing a new kernel automatically but not alerting the user that it's not running, which is one of the things this bug is about, as well as failing to reboot automatically when unattended-upgrades has been set to do so.

[Seth Arnold (seth-arnold)]

(I'm not on the livepatch team)

> How does one configure the system so that it automatically reboots when required, except when livepatch is sufficient?

Livepatches for kernel flaws come after updated kernel packages: if you always reboot into a new kernel the day the kernel is delivered, you'll never install livepatches.

> If a user wants to rely on Livepatch entirely (or 'until further notice'), unattended-upgrades can be set to blacklist automatic kernel upgrades.

I would like to discourage this. Livepatch addresses only issues that (a) have had a CVE number assigned (b) have been scored as high or critical by our triage process. A lot of kernel security issues never get CVE numbers assigned. When we import new stable kernels from upstream developers, there may be security fixes that aren't actually called out as security fixes anywhere.

While it might seem wasteful to download a few hundred megabytes and rebuild an initramfs every two weeks regardless if you plan to reboot into it, unexpected reboots *do* happen, and it would be nice to get the most up-to-date kernel when that happens.

(Maybe an organization would rather have the entire fleet move along in a cadence, or only deploy kernels once they've been vetted through A/B testing or a staging environment, but that's an entirely separate choice to make. These organizations may disable unattended-upgrades entirely, or own their own repositories and control when packages are released into it, etc.)

Thanks

[mev (mev1)]

> Livepatches for kernel flaws come after updated kernel packages: if you always reboot into a new kernel the day the kernel is delivered, you'll never install livepatches.

Not always, and that's a problem with the current motd warning when Livepatch is operational, as described above. Last month (July 2023), for example, the Livepatch came before the availability of the kernel upgrade, at least on the Azure variant kernel (and possibly others).

[goldengecko (goldengecko)]

Regardless I'm not sure if this provides any additional information/context, I faced this issue during using Cloudron.io as a paying customer (see forum thread https://forum.cloudron.io/topic/9720/reboot-for-security-updates-broken-on-livepatch/) on our Ubuntu 22 LTS server. So the current bug/change in behavior has a productive impact: Due to this, Cloudron doesn't detect anymore that a reboot is required for patching the OS and thus no notification is shown to the admin of the Cloudron instance. The workaround is to logon via shell onto the server and see the reboot hint in the welcome message.

[Robie Basak (racb)]

See also bug 1998947.

[Christian Ehrhardt  (paelzer)]

FYI I'm trying to bring together all the people being involved with the last two changes in that area to avoid fixing this one case but thereby breaking yet another use-case. But it starts rather slowly ... :-/

[Steve Langasek (vorlon)]

There has been a discussion between the several engineering teams involved regarding a path forward on this bug. It's acknowledged that this is a regression which breaks the statement of user intent expressed by setting the Unattended-Upgrade::Automatic-Reboot option. However, this is an uncommon configuration (if you are configuring your system to automatically reboot on kernel upgrade, you are getting very little value out of livepatch on top of that since in the common case the kernel deb is made available to the system before the corresponding livepatch) so most users are not expected to be using both in combination. And we have agreed that we should be getting richer information from canonical-livepatch about the security bug coverage of available livepatches vs kernel debs that will allow a more nuanced decision about whether to tell the user a reboot is required. This work is planned for the 24.04 development cycle and we believe will cover this regression as well; so a targeted fix for this issue is not currently a priority.

[Julian Andres Klode (juliank)]

Dropping the rls-ff-incoming in favor of adding a task (and one for jammy) so it is out of view until nn cycle.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Benjamin Drung (bdrung)]

From IRC meeting:

<vorlon> that one should be taken for noble, but there's no action for Foundations at the moment because there's a dependency on changes to livepatch to expose the necessary information about whether a reboot is required
<vorlon> I think we should go ahead and target this