Remove oem-flavour.cfg for the OEM kernel retirement

[Shih-Yuan Lee]

[ Impact ]

 * This issue affects all Ubuntu OEM preloaded systems that are still utilizing the OEM kernel, particularly those equipped with NVIDIA graphics drivers.
 * Systems may fail to boot properly due to an outdated OEM kernel conflicting with newer NVIDIA drivers, leading to a black screen on reboot.

[ Test Plan ]

* To verify the fix, ensure that the file `/etc/default/grub.d/oem-flavour.cfg` is successfully removed from the affected system after applying the update.
* After the system reboots, check that it boots into the general HWE kernel rather than the OEM kernel.

* Idempotency Testing:
  - Perform the upgrade multiple times on the same system to verify that the upgrade process is idempotent, ensuring that repeated upgrades do not result in unexpected behavior or changes.

* Non-OEM System Testing:
  - Test the upgrade on a system without OEM kernels to ensure that the upgrade does not affect systems that are not preloaded with the OEM kernel.
  - Confirm that the system continues to function normally and boots into the correct kernel after the update.

[ Where problems could occur ]

 * Removing `/etc/default/grub.d/oem-flavour.cfg` may cause systems relying on the OEM kernel for specific hardware compatibility to lose access to certain OEM-specific optimizations.
 * There is a low risk that after removing the OEM kernel, some hardware features may not function as expected if the general HWE kernel does not fully support them.

[ Other Info ]

 * This patch is targeted only for Ubuntu 22.04 and its associated OEM kernels.
 * The issue is specific to systems using NVIDIA drivers in conjunction with an outdated OEM kernel, which conflicts with updates to the HWE kernel and NVIDIA packages.

1) The release of Ubuntu you are using, via 'lsb_release -rd' or System -> About $ lsb_release -rd
Description: Ubuntu 22.04.4 LTS
Release: 22.04
$ ubuntu-report show | grep DCD
    "DCD": "canonical-oem-sutton-jammy-amd64-20240606-874"

2) The version of the package you are using, via 'apt-cache policy pkgname' or by checking in Software Center
$ apt-cache policy ubuntu-drivers-common
ubuntu-drivers-common:
  Installed: 1:0.9.6.2~0.22.04.6
  Candidate: 1:0.9.6.2~0.22.04.7
  Version table:
     1:0.9.6.2~0.22.04.7 500
        500 http://tw.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
 *** 1:0.9.6.2~0.22.04.6 100
        100 /var/lib/dpkg/status
     1:0.9.6.1 500
        500 http://tw.archive.ubuntu.com/ubuntu jammy/main amd64 Packages

3) What you expected to happen

 * The system should seamlessly upgrade NVIDIA drivers and the HWE kernel without breaking the boot process. The system should boot into the latest supported HWE kernel without encountering compatibility issues with the NVIDIA drivers.

4) What happened instead

 * After unattended-upgrades, the system rebooted into the OEM kernel, which is outdated and incompatible with the new NVIDIA drivers. As a result, the screen remained black upon reboot.

We need to remove /etc/default/grub.d/oem-flavour.cfg to avoid the system booting from deprecated OEM kernels.

[Shih-Yuan Lee (fourdollars)]

u@ubuntu:~$ ls -l /etc/default/grub.d/oem-flavour.cfg
lrwxrwxrwx 1 root root 47 Sep 27 16:06 /etc/default/grub.d/oem-flavour.cfg -> /usr/share/oem-sutton-bast-meta/oem-flavour.cfg
u@ubuntu:~$ apt-cache policy ubuntu-drivers-common
ubuntu-drivers-common:
  Installed: 1:0.9.6.2~0.22.04.6
  Candidate: 1:0.9.6.2~0.22.04.8
  Version table:
     1:0.9.6.2~0.22.04.8 500
        500 https://ppa.launchpadcontent.net/fourdollars/experimental/ubuntu jammy/main amd64 Packages
     1:0.9.6.2~0.22.04.7 500
        500 http://tw.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
 *** 1:0.9.6.2~0.22.04.6 100
        100 /var/lib/dpkg/status
     1:0.9.6.1 500
        500 http://tw.archive.ubuntu.com/ubuntu jammy/main amd64 Packages
u@ubuntu:~$ sudo apt install ubuntu-drivers-common
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  apt-file dmraid gir1.2-timezonemap-1.0 gir1.2-xkl-1.0 iperf kpartx kpartx-boot libdebian-installer4 libdmraid1.0.0.rc16 libopenscap8 libregexp-assemble-perl libtimezonemap-data
  libtimezonemap1 msr-tools pastebinit python3-icu python3-pam rdate ubiquity-casper ubiquity-ubuntu-artwork xautomation
Use 'sudo apt autoremove' to remove them.
Suggested packages:
  python3-aptdaemon.pkcompat
The following packages will be upgraded:
  ubuntu-drivers-common
1 upgraded, 0 newly installed, 0 to remove and 195 not upgraded.
Need to get 85.2 kB of archives.
After this operation, 1,024 B of additional disk space will be used.
Get:1 https://ppa.launchpadcontent.net/fourdollars/experimental/ubuntu jammy/main amd64 ubuntu-drivers-common amd64 1:0.9.6.2~0.22.04.8 [85.2 kB]
Fetched 85.2 kB in 2s (48.0 kB/s)
Preconfiguring packages ...
(Reading database ... 202947 files and directories currently installed.)
Preparing to unpack .../ubuntu-drivers-common_1%3a0.9.6.2~0.22.04.8_amd64.deb ...
Unpacking ubuntu-drivers-common (1:0.9.6.2~0.22.04.8) over (1:0.9.6.2~0.22.04.6) ...
Setting up ubuntu-drivers-common (1:0.9.6.2~0.22.04.8) ...
/etc/default/grub.d/oem-flavour.cfg contains oem-sutton-bast-meta and GRUB_FLAVOUR_ORDER=oem. Removing oem-flavour.cfg...
u@ubuntu:~$ ls -l /etc/default/grub.d/oem-flavour.cfg
ls: cannot access '/etc/default/grub.d/oem-flavour.cfg': No such file or directory

[Robie Basak (racb)]

SRU review.

I appreciate the importance of fixing this and the general approach seems OK to me.

0. I do have some difficulty in understand the exact mechanism involved. Is it correct that in *all* cases it's now wrong in Jammy for /etc/default/grub.d/oem-flavour.cfg to point to `oem-somerville*-meta|oem-stella*-meta|oem-sutton*-meta` what about after the user upgrades to Noble? What should happen then? What assurances do we have that removing /etc/default/grub.d/oem-flavour.cfg will result in a still-working kernel? I think it would really help to document the technical mechanisms involved here in more detail so that reviewers and observers can get some more confidence in the fix, but I don't think that part is a blocker for SRU.

1. I understand that this currently only affects Jammy. But it's a general pattern that will eventually happen on every LTS, right? So from the SRU philosophy that we avoid future regressions by fixing the development release first, shouldn't there at least be a bug that tracks a long term solution to the general problem that, when fixed, will stop this problem recurring?

2. I appreciate that the severity of this bug and its ability to regress existing behaviour means that we shouldn't block on such a fix actually landing before we hack a fix for Jammy, but could that bug at least be created so that it is tracked, please?

3. I appreciate that this is both important and that some hack is going to have to go somewhere. But please could you get an ack for this from someone on the desktop team please, as they maintain this package and if we're going to do something as unusual as this then they should probably have input?

4. I think the "&& exit 0" lines are a bug, because they will preclude any #DEBHELPER# snippets from running. It looks like debhelper already has generated code in ubuntu-drivers.postinst so this seems like a real and immediate issue. Probably the easiest is to move this into a function and use `return`, but note that then you'll need to do explicit error checking since `set -e` will no longer work. But also, please add `set -e` anyway as is best practice.

5. Why is a previous debian/changelog entry being changed? If you're correcting a previous mistake then I guess it's subjective as to whether that's OK, but I can't distinguish between that and an inadvertent change. Please could you clarify?

6. Shouldn't you be triggering update-grub after touching /etc/default/grub.d, or is there a mechanism for that already in place?

7. I think some additional testing is warranted: specifically to test idempotency of the upgrade path, as well as correct handling for the common case of a non-OEM install. Please could you add to the Test Plan to test these paths?

Due to point 4, I fairly sure that the current upload needs amending, so I'm rejecting it from the queue.

[Timo Aaltonen (tjaalton)]

I'll answer what I can:

0. the machines are already on the hwe track, and that itself happens only when we consider the hwe kernel to have all the driver fixes etc that the oem kernel carried.. so upgrade to noble should not break things

1./2. yeah I bet this will be discussed at the incoming sprint, but the solution might not be in a distro package but something that is shipped via the oem archive. That said, the 'OEM Priority Project' which is marked here as well, could maybe carry that

3. the package is actually maintained by the kernel team, traditionally by the NVIDIA driver maintainer, and the change has landed to the jammy git branch

5. hmmm might be something caused by the git and archive being out of sync with that entry.. will check for the next iteration

[Robie Basak (racb)]

I thought of a further edge case:

8. What happens if reimaging from the OEM image, and then upgrading to the updates pocket all at once, including the update that triggered this bug together with this for it? Which postinst will run first? If not deterministic, and this one runs first, will that cause a problem? Do we need a Breaks clause for ordering purposes? Or is it OK to remove /etc/default/grub.d/oem-flavour.cfg _before_ the bug is triggered?

[Robie Basak (racb)]

> 3. the package is actually maintained by the kernel team, traditionally by the NVIDIA driver maintainer, and the change has landed to the jammy git branch

Shouldn't the kernel team be a bug subscriber then? I only see the desktop team. Who monitors bugs against this package?

[Shih-Yuan Lee (fourdollars)]

> 8. What happens if reimaging from the OEM image, and then upgrading to the updates pocket all at once, including the update that triggered this bug together with this for it? Which postinst will run first? If not deterministic, and this one runs first, will that cause a problem? Do we need a Breaks clause for ordering purposes? Or is it OK to remove /etc/default/grub.d/oem-flavour.cfg _before_ the bug is triggered?

If OEM metapackage postinst runs first, it will create the symbolic link of /etc/default/grub.d/oem-flavour.cfg. And then ubuntu-drivers-common postinst will remove it.
If ubuntu-drivers-common postinst first, there is nothing happening. But OEM metapackage postinst will still create the symbolic link of /etc/default/grub.d/oem-flavour.cfg.

However the content in /etc/default/grub.d/oem-flavour.cfg for the later OEM images from right now or OEM metapackages in OEM archives for jammy should only contain 'GRUB_FLAVOUR_ORDER=generic'.

If /etc/default/grub.d/oem-flavour.cfg exists after the system upgrade, the content should be 'GRUB_FLAVOUR_ORDER=generic' and the system will boot from the latest generic hwe kernel.
If /etc/default/grub.d/oem-flavour.cfg doesn't exist after the system upgrade, the system will boot from the latest kernel with the biggest version, i.e. generic hwe kernel.

[Shih-Yuan Lee (fourdollars)]

> 7. I think some additional testing is warranted: specifically to test idempotency of the upgrade path, as well as correct handling for the common case of a non-OEM install. Please could you add to the Test Plan to test these paths?

The patch is designed to activate only when /etc/default/grub.d/oem-flavour.cfg exists. It should be nothing happening if /etc/default/grub.d/oem-flavour.cfg doesn't exist.

[Shih-Yuan Lee (fourdollars)]

> 6. Shouldn't you be triggering update-grub after touching /etc/default/grub.d, or is there a mechanism for that already in place?

The issue itself is triggered by unattended-upgrades to install the generic hwe kernel and NVIDIA packages so it will execute update-grub for sure, but the order is unknown so we shall make it trigger updata-grub in or after ubuntu-drivers-common postinst. I will improve this in the following patch.

[Shih-Yuan Lee (fourdollars)]

> 5. Why is a previous debian/changelog entry being changed? If you're correcting a previous mistake then I guess it's subjective as to whether that's OK, but I can't distinguish between that and an inadvertent change. Please could you clarify?

I just get the Debian source package by `pull-lp-source ubuntu-drivers-common jammy` and then generate the debdiff.
Is it because of https://github.com/canonical/ubuntu-drivers-common/commit/cde7ffd08f703f2b0cce5c099bb7b95ae56f457b?

[Shih-Yuan Lee (fourdollars)]

> 4. I think the "&& exit 0" lines are a bug, because they will preclude any #DEBHELPER# snippets from running. It looks like debhelper already has generated code in ubuntu-drivers.postinst so this seems like a real and immediate issue. Probably the easiest is to move this into a function and use `return`, but note that then you'll need to do explicit error checking since `set -e` will no longer work. But also, please add `set -e` anyway as is best practice.

I will improve it in the following patch.

[Timo Aaltonen (tjaalton)]

5 is fixed in the new version, it was just a mismatch with git vs archive, no actual code change in git to match the diff

For the future-proofness case, I'm thinking that there should be a metapackage in main which installs the oem kernel and the grub symlink. Then once the oem->hwe kernel migration happens on the kernel side, the metapackage would drop the symlink and update it's dependencies. This would be something unattended-upgrades could be able to handle.

[Timo Aaltonen (tjaalton)]

note that I reused the .8 version, as the previous upload was rejected

[Andreas Hasenack (ahasenack)]

9) dpkg -S can return multiple packages in one line, and also over multiple lines, depending on what matches.

Example for one line with multiple packages:
$ dpkg -S /etc/grub.d
tuned, fwupd, grub-common, memtest86+: /etc/grub.d

And multiple lines:
$ dpkg -S grub.d
grub-common: /etc/grub.d/20_linux_xen
grub-common: /etc/grub.d/10_linux_zfs
memtest86+: /etc/grub.d/20_memtest86+
fwupd: /etc/grub.d/35_fwupd
grub-common: /etc/grub.d/25_bli
grub-common: /etc/grub.d/05_debian_theme
grub-common: /etc/grub.d/README
grub-common: /etc/grub.d/40_custom
grub-common: /etc/grub.d/30_os-prober
grub-common: /etc/grub.d/00_header
grub-common: /etc/grub.d/10_linux
grub-common: /etc/grub.d/41_custom
tuned, fwupd, grub-common, memtest86+: /etc/grub.d
tuned: /etc/grub.d/00_tuned
grub-common: /etc/grub.d/30_uefi-firmware
grub-common: /etc/default/grub.d

Is the symlink target of /etc/default/grub.d/oem-flavour.cfg guaranteed to be owned by a single package in any ubuntu system?

10) How sure are we that the HWE kernel has the exact same functionality of the oem kernel? Why not update the OEM kernel to match the nvidia drivers?

11) Shouldn't there be a reboot notification in this update?

12) What is it that makes it a sure bet that removing that oem-flavor symlink will leave the system with a bootable kernel? Or in other words, what is guaranteeing that the HWE, or any other kernel, is installed?

13) Why is this being done in ubuntu-drivers-common, and not an oem-specific package? Perhaps even the one that puts that symlink in place?
ubuntu-drivers-common is seeded all over the place, and many (all?) users will get this update. Or is it that just nvidia users should have this package installed? I have it installed on my laptop, which has no nvidia card, btw.

14) Out of curiosity, I installed oem-somerville-lapras-meta 22.04ubuntu9 in a jammy vm. In its oem-somerville-lapras-meta.postinst we see:

case "$1" in
    configure)
        mkdir -p /etc/default/grub.d/
        ln -sf /usr/share/oem-somerville-lapras-meta/oem-flavour.cfg /etc/default/grub.d/oem-flavour.cfg
        if [ -e /boot/grub/grub.cfg ] && command -v update-grub; then
            update-grub
        fi
    ;;
esac

So if oem-somerville-lapras-meta is reinstalled, or upgraded, won't it just create the symlink again? Wouldn't this be the best place to a) remove the symlink; b) not create it after some specific version?

15) Where is GRUB_FLAVOUR_ORDER used/defined? (This is more of a curiosity question, as I haven't encountered this variable before). Is it used by update-grub?

[Timo Aaltonen (tjaalton)]

9) yes, there is a single oem-$project-$platform metapackage installed

10) the migration of oem to hwe has been done already by a separate process, so yes, we are sure

11) I don't see a reason to make it any more complicated

12) the usual handling of metapackage upgrades oem -> hwe, already verified before

13) u-d-c is owned by the kernel team, there is no oem-specific package in main, which is required for unattended-upgrades to work

14) we can't use oem metapackages, because u-a don't install anything outside security

15) no idea

[Timo Aaltonen (tjaalton)]

more about 14)

the oem metapackages will surely get updated too, but we shouldn't wait for that to happen to block this hotfix for a rather academic scenario

[Shih-Yuan Lee (fourdollars)]

15) We added https://git.launchpad.net/~ubuntu-core-dev/grub/+git/ubuntu/commit/?h=focal&id=2310fddfe741bf77f8721675a679478d0b6a5613 since focal that will order the kernel flavour we specify first in the GRUB boot entries.

[Andreas Hasenack (ahasenack)]

> However the content in /etc/default/grub.d/oem-flavour.cfg for the later OEM images from right now or OEM
> metapackages in OEM archives for jammy should only contain 'GRUB_FLAVOUR_ORDER=generic'.

I actually confirmed this, by installing oem-somerville-lapras-meta=22.04~ubuntu1 on jammy (that package comes from jammy-updates, and is not available in any other pocket, not even release). That did not create the symlink in /etc/default/grub.d yet.

And 22.04ubuntu9 is available in the dell repository:

$ apt-cache policy oem-somerville-lapras-meta
oem-somerville-lapras-meta:
  Installed: 22.04~ubuntu1
  Candidate: 22.04ubuntu9
  Version table:
     22.04ubuntu9 500
        500 http://dell.archive.canonical.com jammy/somerville-lapras amd64 Packages
 *** 22.04~ubuntu1 500
        500 http://br.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        100 /var/lib/dpkg/status

Updating to 22.04ubuntu9 gives me:
$ cat /etc/default/grub.d/oem-flavour.cfg
# This file is automatically generated by oem-somerville-lapras-meta, and changes will be overriden
GRUB_FLAVOUR_ORDER=generic

looks good then.

[Andreas Hasenack (ahasenack)]

> more about 14)
>
> the oem metapackages will surely get updated too, but we shouldn't wait for that to happen to block this
> hotfix for a rather academic scenario

Agreed, let's just not forget about it should the oem package ever get another update.

[Andreas Hasenack (ahasenack)]

Hello Shih-Yuan, or anyone else affected,

Accepted ubuntu-drivers-common into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ubuntu-drivers-common/1:0.9.6.2~0.22.04.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted ubuntu-drivers-common (1:0.9.6.2~0.22.04.8) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

update-notifier/3.192.54.8 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#ubuntu-drivers-common

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Atlas Yu (pseudoc)]

After enable the proposed channel and install ubuntu-drivers-common=1:0.9.6.2~0.22.04.8, the oem-flavour.cfg (a symbolic link created by the OEM meta package) get removed, and the machine can boot into:

6.8-generic kernel when both 6.8-generic and 6.5-oem kernel are installed.
6.5-oem kernel when 6.5-oem kernel is installed.

[Atlas Yu (pseudoc)]

After enable the proposed channel and install ubuntu-drivers-common=1:0.9.6.2~0.22.04.8 on non-OEM system machines. The machines can boot into the 6.8-generic kernel, and everything works fine.

[Atlas Yu (pseudoc)]

Idempotency testing results are also good on both OEM/non-OEM systems.

[Timo Aaltonen (tjaalton)]

the autopkgtest regression on armhf is a false alarm, just failing to connect to lp

[Andreas Hasenack (ahasenack)]

autopkgtests green so far

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-drivers-common - 1:0.9.6.2~0.22.04.8

---------------
ubuntu-drivers-common (1:0.9.6.2~0.22.04.8) jammy; urgency=medium

  [Shih-Yuan Lee]
  * debian/ubuntu-drivers-common.postinst: Remove
    /etc/default/grub.d/oem-flavour.cfg for the OEM kernel retirement.
    (LP: #2083657)

 -- Kuba Pawlak <email address hidden> Mon, 07 Oct 2024 16:57:26 +0200

[Timo Aaltonen (tjaalton)]

The verification of the Stable Release Update for ubuntu-drivers-common has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Bin Li (binli)]

The new version is just in updates, we still need help to let it in jammy-security.

$ apt-cache policy ubuntu-drivers-common
ubuntu-drivers-common:
  Installed: (none)
  Candidate: 1:0.9.6.2~0.22.04.8
  Version table:
     1:0.9.6.2~0.22.04.8 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
     1:0.9.6.1 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

[Steve Beattie (sbeattie)]

THis was pocket copied to jammy-security on 2024-10-30 after verification that the dependencies were satisfiable for people without jammy-updates enabled: https://launchpad.net/ubuntu/+source/ubuntu-drivers-common/1:0.9.6.2~0.22.04.8

Thanks!