efivars file system missing in Ubuntu 22.04 real-time kernel

[Alex Wang]

In Ubuntu 22.04 generic kernel like 5.15.0-23, efivars file system is mounted and is visible in the output of mount command, however in Ubuntu 22.04 real-time kernel like 5.15.0-1005-realtime or 5.15.0-1007-realtime, efivars file system is missing. Intel SGX feature relies on efivars file system to function, could u please investigate this issue? Thanks.

---

In ubuntu, multiple things rely on reliable access to efivars (read-only) and to have ability to manipulate them too (read-write). Thus imho we should revert the v5.15 patch that turns efivars by default; and in later series update annotation to keep it on, even under realtime.

Things sort of work on boot, as shim fallback app (fb*.efi) parsses, loads and sets initial boot variables. However subsequent updates to our bootloaders (shim, grub, nullboot, snapd) do not know if they are set, if they are correct, or if they can be used. Functionality that is missing on such systems is then thus inability to install fw updates with fwupd, inatibility to boot into firmware setup (systemctl reboot --firmware-setup), and inability to predict measurements to predict sealing policies with new updates in case of TPM based sealed secrets (i.e. UC based FDE, systemd based secrets, SGX, etc).

I will use this bug report to address this by default. Users that are concerned about userspace/OS accessing and using efivars during maintainance operations (package upgrades) or during runtime otherwise (arbitrary calls to bootctl for example), should consider getting hardware that has realtime aware EFI implementation, or modify their classic or core systems to disable efi runtime services by opting-out of efivars.

[Ubuntu Foundations Team Bug Bot (crichton)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Libera.chat.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1970077/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1970077

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Alex Wang (alexwang-bkc)]

~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=98812568k,nr_inodes=24703142,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=19773308k,mode=755,inode64)
/dev/mapper/ubuntu--vg-ubuntu--lv on / type ext4 (rw,relatime,stripe=64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=138479)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
none on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/var/lib/snapd/snaps/core20_1405.snap on /snap/core20/1405 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/lxd_22923.snap on /snap/lxd/22923 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/var/lib/snapd/snaps/snapd_15534.snap on /snap/snapd/15534 type squashfs (ro,nodev,relatime,errors=continue,x-gdu.hide)
/dev/sda2 on /boot type ext4 (rw,relatime,stripe=64)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,nodev,noexec,relatime,size=19773308k,mode=755,inode64)
nsfs on /run/snapd/ns/lxd.mnt type nsfs (rw)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=19773304k,nr_inodes=4943326,mode=700,inode64)

[Alex Wang (alexwang-bkc)]

# uname -r
5.15.0-1009-realtime

[Alex Wang (alexwang-bkc)]

we have tried the latest available RT kernel 5.15.0-1009-realtime, it's still not there. Could u please help check? Let me know if you need other logs. Thanks.

[Cindy Goldberg (cindykgo)]

The Real-time kernel (beta) is available through UA.

Please signup on https://ubuntu.com/realtime-kernel and you will be guided through the process.
This process will get the user a free Ubuntu Advantage account and access to the RTK repos.
Enabling RT via ua enable realtime-kernel will add the new PPA under UA, install and configure the necessary RT modules (e.g. kernel params), and provide the necessary updates. Each package supported in the RT PPA will have a non-RT version available in the archive (special config only in RT-enabled archives).
Both the base (i.e. generic) kernel in the public archive and the RT-enabled kernel in the separate repo will be regularly updated as per the standard 3-week SRU cycle.

[Joseph Salisbury (jsalisbury)]

One of the upstream real-time patches explicitly disables access to efi vars because some operations can cause latency spikes.

Here is the specific commit:
https://git.launchpad.net/~canonical-kernel-rt/ubuntu/+source/linux-realtime/+git/jammy/commit/?id=9d3a4cc9721d9f48075dbaf397712974b0ef9af1

If you really need to have access to the efi vars you can boot with efi=runtime, but that might affect the RT behavior of the kernel.

[Alex Wang (alexwang-bkc)]

As a long term solution to get better compatibility in Ubuntu22.04 real-time for SGX feature (which being widely used by Network Platform requirement, could we request Canonical to remove this patch or implement an alternative change?

The basic concerns or reasons behind are:
1. If just simply disabled efiv_ar as the default setting in Ubuntu22.04 real-time environment, that causes SGX feature cannot be used.
2. The original commit was based on test on an AMD ARM chip, On IA X86 systems, it’s breaking something that was actually working in the first place.

Please help to weight in and feed back.

Tons of thanks.

[Joseph Salisbury (jsalisbury)]

Here is the commit that disables EFI, which we have applied to our RT kernel:
https://git.launchpad.net/~canonical-kernel-rt/ubuntu/+source/linux-realtime/+git/jammy/commit/?id=9d3a4cc9721d9f48075dbaf397712974b0ef9af1

This commit is applied as one of the real-time patches:
0021-efi-Disable-runtime-services-on-RT.patch:Subject: [PATCH 021/166] efi: Disable runtime services on RT

I did some digging around and see that this patch is not applied by the upstream real-time patches for a 5.16 or newer kernel. I reviewed the upstream real-time mailing list and found this thread:
https://www.spinics.net/lists/linux-rt-users/msg25797.html

However, this patch does not seem to have ever been applied to mainline and it is not in any of the upstream patch sets.

That ML thread has been dormant for a while. We can ping it to see what the status is.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1970077

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Joseph Salisbury (jsalisbury)]

I will revert the disabling of efivars in the next release of the Ubuntu real-time kernel.

[Joseph Salisbury (jsalisbury)]

efivars will be enabled by default in the RT kernel as of version 5.15.0-1045.50.