initramfs need to mount efivarfs because kernel 6.0 deprecated 'efivars' sysfs interface

[Cyrus Lien]

[ Impact ]

kernel 6.0 deprecated efivars sysfs interface [1]. For Intel VROC RAID, mdadm needs initramfs to mount efivarfs instead.

[1] The commit:
commit 0f5b2c69a4cbe4166ca24b76d5ada98ed2867741
Author: Ard Biesheuvel <email address hidden>
Date: Mon Jun 20 13:34:03 2022 +0200

    efi: vars: Remove deprecated 'efivars' sysfs interface

[ Test Plan ]

1. Install mdadm (4.2-0ubuntu2) from -proposed.
2. Adding test script init-top/01_check_mount to list all mounted filesystems to initramfs.debug.
3. Adding test script init-bottom/zz_check_mount to list all mounted filesystems to initramfs.debug.
3. update-initramfs -u
4. Append "debug" to GRUB_CMDLINE_LINUX_DEFAULT then sudo update-grub.
5. Reboot and check /var/run/initramfs/initramfs.debug if the boot script 00_mount_efivarfs will be executed before udev.

[Test script] 01_check_mount and zz_check_mount are identical.

#!/bin/sh

PREREQ=""
prereqs()
{
 echo "$PREREQ"
}
case $1 in
# get pre-requisites
prereqs)
 prereqs
 exit 0
 ;;
esac

echo "$(mount)"

[ Where problems could occur ]

Not sure if there any other tools/utilities also need to mount efivarfs as early as mdadm but the probability of file conflict should be very low.
Also, there are no impact mounting efivarfs multiple times.
mount: /sys/firmware/efi/efivars: efivarfs already mounted on /sys/firmware/efi/efivars.

[ Scope ]

Jammy, Kinetic, Lunar, Mantic

[ Other Info ]

The private bug link https://bugs.launchpad.net/somerville/+bug/1990231

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "mdadm_4.2-0ubuntu2.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Olivier Gayot (ogayot)]

@cyruslien what series would you need the patch to be applied to?

If this is for a SRU, would you mind filling the SRU template? Thanks!

[Cyrus Lien (cyruslien)]

@Olivier,

This is patch v2 and the target series is jammy, thanks.

[Benjamin Drung (bdrung)]

Arch Linux solved this issue by mounting efivarfs when mounting all other paths: https://github.com/archlinux/mkinitcpio/commit/70511e5ccb7e2500213e04173a70c067a7c9aa35. mount_table in systemd contains efivarfs as well.

So I prefer putting the mounting code into initramfs-tools instead.

[Cyrus Lien (cyruslien)]

Ok, filed initramfs-tools SRU lp:1994936. Thanks!

[William Wilson (jawn-smith)]

Marking as invalid since a separate bug has been created against the correct package.

[Steve Langasek (vorlon)]

I disagree that this belongs in initramfs-tools. Initramfs-tools should not be responsible for mounting every deprecated filesystem that some other package providing initramfs hooks requires. This is mdadm's problem to fix, NOT initramfs-tools (and then mdadm should stop mounting it as soon as it is fixed to use the not-deprecated firmware interface).

[Cyrus Lien (cyruslien)]

Although mdadm 4.2+20230223-1 mentioned that debbug #1003352 was fixed, efivarfs still has not been mounted during boot up in my test. So we still need this sauce patch.

[Steve Langasek (vorlon)]

4.2+20230223-1 is only in Debian experimental anyway.

[Steve Langasek (vorlon)]

The patch looks good. Have you confirmed that it doesn't regress things on *pre*-6.0 kernels, which already mount this vfs?

[Cyrus Lien (cyruslien)]

Yes, it wouldn't break things.
Because, even without the patch, systemd still auto-mount the new interface after chroot from initrd. So, on *pre*-6.0 kernels, both new interface (/sys/firmware/efi/efivars) and deprecated interface (/sys/firmware/efi/vars) will be mounted.
The difference that the patch made is mounting efivarfs earlier.

[Steve Langasek (vorlon)]

I'm asking whether you've tested this patch with pre-6.0 kernels. It *should* work, but there could be non-obvious failure modes, such as busybox's mount implementation failing when mounting the same filesystem on the same mountpoint where it is already mounted; or problems when trying to unmount / move mounts when transitioning from the initramfs to the rootfs.

[Cyrus Lien (cyruslien)]

Thank you for your reminder.
Here is my test result.

1. Mount same mountpoint:
Test method: Adding init-top/01_mount_efivarfs_twice to mount efivarfs again.
Result: Indeed, it failed to mount at second time, and "Device or resource busy" error message shows up but that doesn't affect the boot scripts keep going also doesn't impact mdadm.
Log: initramfs.debug

2. Unmount before transitioning to rootfs:
Test method: Adding init-bottom/zz_unmount_efivarfs to unmount efivarfs just before transition.
Result: There is no problem to unmount efivarfs at this stage.
Log: initramfs.debug

3. Also compared the file attributes in efivarfs. The attributes are identical no matter efivarfs is mounted in initrd or by systemd.
Log: lsattr-efivars-mount-by-systemd.log, lsattr-efivars-mount-in-initrd.log

Please find the test scripts and logs in the attached tarball. Thank you.

[Timo Aaltonen (tjaalton)]

please prepare an upload for lunar, then kinetic and jammy

[Steve Langasek (vorlon)]

There is an upload in the lunar queue.

[Timo Aaltonen (tjaalton)]

Indeed, spotted it now.

[Robie Basak (racb)]

> kernel 6.0 deprecated efivars sysfs interface [1]

[...]

> I disagree that this belongs in initramfs-tools. Initramfs-tools should not be responsible for mounting every deprecated filesystem that some other package providing initramfs hooks requires. This is mdadm's problem to fix, NOT initramfs-tools (and then mdadm should stop mounting it as soon as it is fixed to use the not-deprecated firmware interface).

This seems concerning to me. For the development release I agree with Steve. For HWE kernel SRUs in existing stable releases, shouldn't we be maintaining compatibility to all of userspace, rather than changing behaviour for all packages? What if a user is relying on this interface in their own software? They would reasonably expect an HWE kernel not to break that. So I think it follows that for SRU purposes we should maintain userspace in the same state whether or not you booted an HWE kernel. So that should be done somewhere that is guaranteed that all users will receive the fix, and surely that's not mdadm?

[Andreas Hasenack (ahasenack)]

Writing up a summary, mostly for my own understanding, and then some questions at the end.

Old mechanism: /sys/firmware/efi/vars <-- NOT a mount point: normal sysfs interface for kernel < 6.x. GONE in kernel 6.x

New mechanism: /sys/firmware/efi/efivars <-- efivarfs mountpoint. Required for kernel >= 6.x. Unsure if there is another way to get efi variables in /sys without this mountpoint on kernel 6.x. Apparently not.

I have a kinetic box that has both of the above, and it's running kernel 5.x (but after boot: I didn't check the initramfs env). So userspace, after boot at least, on kinetic/kernel 5.x, is already exposed to the new efivarfs mountpoint.

Lunar obviously only has the efivarfs mountpoint mechanism available (kernel 6.x) after boot.

The two mechanisms are different interfaces indeed, it's not just a path change.

mdadm tries both[1]: first the new, then the old. So it's an example of software that could behave differently once the new mountpoint is available in the initramfs environment.

What we need here is for the new mechanism to be mounted in the initramfs environment under kernel 6.x (which is the "if [ -d /sys/<newpath> ]" check in the initramfs script snippet provided in this update). This is not "mounting every deprecated filesystem", but the opposite: it's mounting the "new" filesystem inside the initramfs. Before, with kernel 5.x, we would have the sysfs interface present, whether we needed it or not. It just comes with /sys mounted. Now we would be just continuing that behavior if we always mounted efivarfs in the initramfs all the time.

For SRUs, what this change would do is to mount the new pseudo filesystem, so apps would have two ways to check the efi variables: the old sysfs-interface one, or the new efivarfs one. And as shown above, those two are already avaiable in a kinetic box at runtime. But not in the initramfs environment.

The change the SRU would introduce is that an application like mdadm that checks first for the new efivarfs mountpoint, and if that fails, falls back to the old sysfs interface, would from now on actually find something in that first attempt, and could then change behavior, and run new code paths that would have bugs we haven't seen yet. So maybe that's an argument for making this change only in mdadm for SRUs, instead of in initramfs.

So, I pose these questions:
- for lunar, which is kernel 6.x only, shouldn't this change be in initramfs indeed, as proposed earlier?
- for non-lunar stable releases, right now there is nothing broken, is there? It will be broken once a 6.x hwe kernel starts becoming available (unless it is already) and users install and boot into it. And a non-bootable system is a bad thing, so I can see us trying to prepare for it in advance
- are there other packages besides mdadm which would need efi vars available in the initramfs environment? Perhaps lvm?

1. https://git.launchpad.net/ubuntu/+source/mdadm/tree/platform-intel.c?h=applied/ubuntu/lunar-devel#n560

[Steve Langasek (vorlon)]

Thanks for this important clarification, Andreas. I misremembered, thinking that both /sys/firmware/efi/vars/ and /sys/firmware/efi/efivars/ were mount points, and believed that this was about mounting the *deprecated* path, rather than about mounting the new path so that it's available for use in the initramfs.

As to whether this then belongs in initramfs-tools rather than mdadm, I still think mdadm is the right place even though this isn't a deprecated filesystem, because it's unusual for anything in the initramfs to need access to firmware variables - VROC is the only instance I'm aware of. So I expect this only affects mdadm, and we should keep the code close together.

Regarding risk of regression from switching to use the new interface instead of the old interface on old kernels: the interfaces of the two paths, while not identical, are similar enough that I do not expect any subtle issues with parsing of the data (it should either work all the time or none of the time). However, if you want to mitigate any possible risk on older kernels, we could adjust the initramfs script to mount efivarsfs ONLY if /sys/firmware/efi/vars is not present. Thoughts?

[Andreas Hasenack (ahasenack)]

> So I expect this only affects mdadm, and we should keep the code close together

Then I agree. If we find other instances where this is needed by other packages in initramfs, we can revisit.

> we could adjust the initramfs script to mount efivarsfs ONLY if /sys/firmware/efi/vars is not present. Thoughts?

That sounds ideal for an SRU.

[Steve Langasek (vorlon)]

> > we could adjust the initramfs script to mount efivarsfs ONLY if
> /sys/firmware/efi/vars is not present. Thoughts?

> That sounds ideal for an SRU.

Ok, reuploaded with this change.

[Timo Aaltonen (tjaalton)]

Hello Cyrus, or anyone else affected,

Accepted mdadm into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mdadm/4.2-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Cyrus, or anyone else affected,

Accepted mdadm into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mdadm/4.2-3ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Cyrus Lien (cyruslien)]

Verified on jammy with 6.1.0-1009-oem kernel, efivrafs was mounted during initramfs boot phase and the issue was fixed.

[Test method]
1. Install mdadm (4.2-0ubuntu2) from -proposed.
2. Adding init-top/01_check_mount to echo mount status to initramfs.debug.
3. Adding init-bottom/zz_check_mount to echo mount status to initramfs.debug.
3. update-initramfs -u
4. Reboot and check

[Test script] 01_check_mount and zz_check_mount are identical.
#!/bin/sh

PREREQ=""
prereqs()
{
 echo "$PREREQ"
}
case $1 in
# get pre-requisites
prereqs)
 prereqs
 exit 0
 ;;
esac

echo "$(mount)"

[Cyrus Lien (cyruslien)]

Verified on jammy with 5.17.0-1030-oem kernel, efivrafs was not mounted during initramfs boot phase as expected.

[Cyrus Lien (cyruslien)]

Verified on kinetic with 6.1.0-1011-oem kernel, efivrafs was mounted during initramfs boot phase and the issue was fixed.

[Cyrus Lien (cyruslien)]

Verified on kinetic with 5.19.0-42-generic kernel, efivrafs was not mounted during initramfs boot phase as expected.

[Andreas Hasenack (ahasenack)]

I just noticed the lunar upload of mdadm is still in unapproved. I'll check it out. Then we need a mantic upload.

[Andreas Hasenack (ahasenack)]

Since lunar ships with a 6.x kernel, it doesn't need the check for the old sysfs interface and the new efivarfs mountpoint, unless you want to keep it the same, for consistency.

I.e., currently we have:

kinetic, jammy:
+if [ -d /sys/firmware/efi/efivars ] && ! [ -d /sys/firmware/efi/vars ]; then
+ mount -t efivarfs -o rw,nosuid,nodev,noexec,relatime efivarfs /sys/firmware/efi/efivars
+fi

lunar, mantic:
+if [ -d /sys/firmware/efi/efivars ]; then
+ mount -t efivarfs -o rw,nosuid,nodev,noexec,relatime efivarfs /sys/firmware/efi/efivars
+fi

[Andreas Hasenack (ahasenack)]

Hello Cyrus, or anyone else affected,

Accepted mdadm into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mdadm/4.2-5ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Andreas Hasenack (ahasenack)]

Cyrus, can you please also verify the lunar upload that I just accepted? And then we will need a mantic upload of the same change.

[Steve Langasek (vorlon)]

I've copied the mdadm binaries forward to mantic-proposed.

[Cyrus Lien (cyruslien)]

Verified on lunar with 6.2.0-20-generic kernel, efivrafs was mounted during initramfs boot phase.

[Cyrus Lien (cyruslien)]

Verified on mantic with 6.2.0-21-generic kernel, efivrafs was mounted during initramfs boot phase.

[Robie Basak (racb)]

@Cyrus please could you confirm what steps you performed to verify these? Did you follow the agreed Test Plan? It sounds like you followed different steps?

[Cyrus Lien (cyruslien)]

@Robie, Checking initramfs.debug should be a more reliable way to check if the fix works instead of the old test plan which only checks if boot script 00_mount_efivarfs exists. So I update the test plan and apologize for the confusion.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mdadm - 4.2-5ubuntu2

---------------
mdadm (4.2-5ubuntu2) lunar; urgency=medium

  * Add scripts/init-top/00_mount_efivarfs, mounting efivarfs. (LP: #1993541)

 -- Cyrus Lien <email address hidden> Tue, 11 Apr 2023 11:22:59 +0800

[Andreas Hasenack (ahasenack)]

I verified the test results and am satisfied that they show the executed planned test case, and that the results are correct.

I agree the updated test case is better (see comment #39)

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions,

There is no SRU freeze ongoing at the moment.

There is no halted phasing on a previous update.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mdadm - 4.2-5ubuntu2

---------------
mdadm (4.2-5ubuntu2) lunar; urgency=medium

  * Add scripts/init-top/00_mount_efivarfs, mounting efivarfs. (LP: #1993541)

 -- Cyrus Lien <email address hidden> Tue, 11 Apr 2023 11:22:59 +0800

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for mdadm has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mdadm - 4.2-3ubuntu2

---------------
mdadm (4.2-3ubuntu2) kinetic; urgency=medium

  * Add scripts/init-top/00_mount_efivarfs, mounting efivarfs. (LP: #1993541)

 -- Cyrus Lien <email address hidden> Tue, 11 Apr 2023 11:22:59 +0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package mdadm - 4.2-0ubuntu2

---------------
mdadm (4.2-0ubuntu2) jammy; urgency=medium

  * Add scripts/init-top/00_mount_efivarfs, mounting efivarfs. (LP: #1993541)

 -- Cyrus Lien <email address hidden> Tue, 11 Apr 2023 11:22:59 +0800