AppArmor fails to start with Yoga UCA libvirt profile on Focal

[Danila Balagansky]

[ Impact ]

AppArmor fails to start with yoga-focal uca libvirt profile

[ Test Plan ]

generate yoga-focal openstack instance
juju ssh nova-compute/0
sudo systemctl restart apparmor
journalctl -xe

# Error message
ct 04 15:55:32 juju-6d4862-apparmorbug-9 apparmor.systemd[94081]: AppArmor parser error for /etc/apparmor.d/usr.sbin.libvirtd in /etc/apparmor.d/usr.sbin.li>
Oct 04 15:55:32 juju-6d4862-apparmorbug-9 apparmor.systemd[94082]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Oct 04 15:55:32 juju-6d4862-apparmorbug-9 audit[94084]: AVC apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="u>
Oct 04 15:55:32 juju-6d4862-apparmorbug-9 apparmor.systemd[94005]: Error: At least one profile failed to load

[ Other Notes ]

On a fully patched Ubuntu Focal with Yoga UCA enabled, after installation of libvirt-daemon-system, restarting apparmor would fail with error:

Aug 31 07:40:52 ubuntu2004.localdomain apparmor.systemd[6335]: Restarting AppArmor
Aug 31 07:40:52 ubuntu2004.localdomain apparmor.systemd[6335]: Reloading AppArmor profiles
Aug 31 07:40:52 ubuntu2004.localdomain apparmor.systemd[6341]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Aug 31 07:40:52 ubuntu2004.localdomain apparmor.systemd[6348]: AppArmor parser error for /etc/apparmor.d in /etc/apparmor.d/usr.sbin.libvirtd at line 29: Invalid capability bpf.
Aug 31 07:40:52 ubuntu2004.localdomain apparmor.systemd[6413]: AppArmor parser error for /etc/apparmor.d/usr.sbin.libvirtd in /etc/apparmor.d/usr.sbin.libvirtd at line 29: Invalid capability bpf.
Aug 31 07:40:52 ubuntu2004.localdomain apparmor.systemd[6418]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Aug 31 07:40:52 ubuntu2004.localdomain apparmor.systemd[6335]: Error: At least one profile failed to load
Aug 31 07:40:52 ubuntu2004.localdomain systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE
Aug 31 07:40:52 ubuntu2004.localdomain systemd[1]: apparmor.service: Failed with result 'exit-code'.
Aug 31 07:40:52 ubuntu2004.localdomain systemd[1]: Failed to start Load AppArmor profiles.

In addition to bpf, perfmon capability, which is also enabled in /etc/apparmor.d/usr.sbin.libvirtd profile, would lead to the same error.

System information:
root@ubuntu2004:~# uname -a
Linux ubuntu2004.localdomain 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu2004:~# dpkg -l libvirt\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==========================================-=======================-============-=============================================================
ii libvirt-clients 8.0.0-1ubuntu7.1~cloud0 amd64 Programs for the libvirt library
ii libvirt-daemon 8.0.0-1ubuntu7.1~cloud0 amd64 Virtualization daemon
ii libvirt-daemon-config-network 8.0.0-1ubuntu7.1~cloud0 all Libvirt daemon configuration files (default network)
ii libvirt-daemon-config-nwfilter 8.0.0-1ubuntu7.1~cloud0 all Libvirt daemon configuration files (default network filters)
un libvirt-daemon-driver-lxc <none> <none> (no description available)
ii libvirt-daemon-driver-qemu 8.0.0-1ubuntu7.1~cloud0 amd64 Virtualization daemon QEMU connection driver
un libvirt-daemon-driver-storage-gluster <none> <none> (no description available)
un libvirt-daemon-driver-storage-iscsi-direct <none> <none> (no description available)
un libvirt-daemon-driver-storage-rbd <none> <none> (no description available)
un libvirt-daemon-driver-storage-zfs <none> <none> (no description available)
un libvirt-daemon-driver-vbox <none> <none> (no description available)
un libvirt-daemon-driver-xen <none> <none> (no description available)
ii libvirt-daemon-system 8.0.0-1ubuntu7.1~cloud0 amd64 Libvirt daemon configuration files
ii libvirt-daemon-system-systemd 8.0.0-1ubuntu7.1~cloud0 all Libvirt daemon configuration files (systemd)
un libvirt-daemon-system-sysv <none> <none> (no description available)
un libvirt-login-shell <none> <none> (no description available)
un libvirt-sanlock <none> <none> (no description available)
ii libvirt0:amd64 8.0.0-1ubuntu7.1~cloud0 amd64 library for interfacing with different virtualization systems
root@ubuntu2004:~# dpkg -l apparmor\*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=======================-=================-============-======================================
ii apparmor 2.13.3-7ubuntu5.1 amd64 user-space parser utility for AppArmor
un apparmor-profiles-extra <none> <none> (no description available)
un apparmor-utils <none> <none> (no description available)

[Heather Lemon (hypothetical-lemon)]

Linking a related LP bug https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1964636

[Christian Ehrhardt  (paelzer)]

I'll have a look for the same in server-backports ppa, but it might be as easy as the old apparmor not knowing about these and failing. If that is true we might need to remove them on the backports.

@Security - is there more to know about these particular features (will they come to focal, is there more to know about it, ...)?

[Christian Ehrhardt  (paelzer)]

Focal:
- apparmor 2.13.3-7ubuntu5.1
- kernel 5.4.0-109-generic
- libvirt:
  a) base 6.0.0-0ubuntu8.16
  b) server-backport 8.0.0-1ubuntu7.2~backport20.04.202210042317~ubuntu20.04.1
  c) UCA Yoga 8.0.0-1ubuntu7.1~cloud0

With none did a restart trigger an issue as reported.
libvirtd is reported to be in enforce mode by aa-status

Something must be different on the affected systems, any idea what it might be?

But also bpf is not present in that file for any of those versions.
For me this is always empty:
  $ grep bpf /etc/apparmor.d/usr.sbin.libvirtd

The reason is (and that explains why it felt known to me) that I have resolved that in march.
 https://git.launchpad.net/~canonical-server/ubuntu/+source/libvirt/commit/?h=backport-libvirt-focal&id=21eb63454433d7b2c2b75f197b7064c96cf7d1e8

Since it is a conffile it might not be updated on upgrades, so I have checked that.
Server backports was fine as expected.

Yoga is indeed still having bpf when purging and re-installing (to force the default conffile in the pachage).

And then I can see it:
Oct 05 16:27:58 f apparmor.systemd[48796]: AppArmor parser error for /etc/apparmor.d/usr.sbin.libvirtd in /etc/apparmor.d/usr.sbin.libvirtd at line 29: Invalid capability bpf.

Oct 05 16:27:58 f apparmor.systemd[48720]: Error: At least one profile failed to load
Oct 05 16:27:58 f systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE
Oct 05 16:27:58 f systemd[1]: apparmor.service: Failed with result 'exit-code'.
Oct 05 16:27:58 f systemd[1]: Failed to start Load AppArmor profiles.

And indeed it is missing here:
https://git.launchpad.net/~ubuntu-cloud-archive/ubuntu/+source/ca-patches/tree/yoga/libvirt.patch

So UCA needs to pick up the patch I referenced above.

[Heather Lemon (hypothetical-lemon)]

When building the debuild there are a few files that got touched like this one,

diff -Nru libvirt-8.0.0/debian/.pc/.quilt_patches libvirt-8.0.0/debian/.pc/.quilt_patches
--- libvirt-8.0.0/debian/.pc/.quilt_patches 1970-01-01 00:00:00.000000000 +0000
+++ libvirt-8.0.0/debian/.pc/.quilt_patches 2022-10-17 15:01:12.000000000 +0000
diff -Nru libvirt-8.0.0/debian/.pc/.quilt_series libvirt-8.0.0/debian/.pc/.quilt_series

I don't believe these should be added, but wanted a second opinion.

Thank You,
Heather Lemon

[John Johansen (jjohansen)]

@paelzer sorry missed your question earlier, yes this is known. The newer capability support for apparmor is being backported to focal. Hopefully we will see the SRU for it this week.

[Edward Hope-Morley (hopem)]

@hypothetical-lemon to get this into focal-yoga it will need to be fixed in Jammy first. As I understand it the problem is focal-specific to either the package needs to be selective on which config it applies based on series or perhaps the uca itself needs fixing to support this on focal.

[Edward Hope-Morley (hopem)]

oh and of course kinetic/zed too

[Heather Lemon (hypothetical-lemon)]

remove the extra quilt .pc additions from the top of the patch

[Heather Lemon (hypothetical-lemon)]

@jjohansen

Are you planning to add this fix to the kernel (focal) as well?

Thank You,
Heather Lemon

[John Johansen (jjohansen)]

@hypothetical-lemon yes we plan to SRU the fix to focal however a fix is only needed in the userspace. The fix is in the apparmor userspace policy compiler, the kernel is just enforcing what the compiler is incorrectly telling it to enforce.

[John Johansen (jjohansen)]

@hypothetical-lemon you can try the fix if you want it is in the ppa https://launchpad.net/~georgiag/+archive/ubuntu/mqueue-sru.

[Heather Lemon (hypothetical-lemon)]

@jjohansen,

I've tested both versions and the fix works.

Is this is the correct place to track the bug? Or is there
another SRU open?

Do I need to delete my patch or should I just leave it there?

Thank You,
Heather Lemon

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Heather Lemon (hypothetical-lemon)]

### VERIFICATION DONE FOCAL ###

# previous apparmor version
apt-cache policy apparmor
package name: apparmor
package version: 2.13.3-7ubuntu5.1
series: Focal
kernel: Linux 5.4.0-136-generic

# before enabling -proposed
generate focal-yoga instance
juju ssh nova-compute/0
# verify no apparmor errors in logs
cat /var/log/syslog | grep Error

# verify apparmor is running
sudo systemctl status apparmor

# trigger error
sudo systemctl restart apparmor

# The apparmor service never successfully restarts
Job for apparmor.service failed because the control process exited with error code.
See "systemctl status apparmor.service" and "journalctl -xe" for details

cat /var/log/syslog

Error messages in syslog:
Jan 11 15:46:14 juju-5c2ee8-appbug-9 apparmor.systemd[52695]: AppArmor parser error for /etc/apparmor.d in /etc/apparmor.d/usr.sbin.libvirtd at line 29: Invalid capability bpf.
Jan 11 15:46:14 juju-5c2ee8-appbug-9 apparmor.systemd[52669]: Error: At least one profile failed to load
Jan 11 15:46:14 juju-5c2ee8-appbug-9 systemd[1]: apparmor.service: Main process exited, code=exited, status=1/FAILURE

### Enable proposed ###

# testing with focal-yoga
Apparmor version tested - 2.13.3-7ubuntu5.2

sudo apt-cache policy apparmor
sudo vim /etc/apt/sources.list
# add -proposed
deb http://nova.clouds.archive.ubuntu.com/ubuntu/ focal-proposed main universe
# save and exit
sudo apt-get update
sudo apt-get upgrade apparmor -y

sudo systemctl restart apparmor
systemctl status apparmor

Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
     Active: active (exited) since Wed 2023-01-11 15:55:19 UTC; 20s ago

tail -n 1000 /var/log/syslog

# no errors are thrown by apparmor
Jan 11 15:54:41 juju-5c2ee8-appbug-9 systemd[1]: Reloading.
Jan 11 15:55:19 juju-5c2ee8-appbug-9 systemd[1]: Starting Load AppArmor profiles...
Jan 11 15:55:19 juju-5c2ee8-appbug-9 apparmor.systemd[66497]: Restarting AppArmor
Jan 11 15:55:19 juju-5c2ee8-appbug-9 apparmor.systemd[66497]: Reloading AppArmor profiles
Jan 11 15:55:19 juju-5c2ee8-appbug-9 kernel: [ 2042.612010] kauditd_printk_skb: 9 callbacks suppressed
Jan 11 15:55:19 juju-5c2ee8-appbug-9 kernel: [ 2042.612013] audit: type=1400 audit(1673452519.139:106): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe" pid=66503 comm="apparmor_parser"
Jan 11 15:55:19 juju-5c2ee8-appbug-9 kernel: [ 2042.612022] audit: type=1400 audit(1673452519.139:107): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="nvidia_modprobe//kmod" pid=66503 comm="apparmor_parser"
Jan 11 15:55:19 juju-5c2ee8-appbug-9 kernel: [ 2042.612179] audit: type=1400 audit(1673452519.139:108): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=66502 comm="apparmor_parser"
Jan 11 15:55:19 juju-5c2ee8-appbug-9 kernel: [ 2042.612183] audit: type=1400 audit(1673452519.139:109): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=66502 comm="apparmor...

[Edward Hope-Morley (hopem)]

apparmor is not provided in the Ubuntu Cloud Archive so we can remove that target from this bug as it is a little misleading. The fix has been released to Focal and Jammy hence why deploying Openstack Yoga on either of those (using focal-yoga uca or jammy-updates) gets you the fixed apparmor. Hope that makes sense.