AppArmor DENIES reading of /sys/devices/system/cpu/possible

[Marius Vollmer]

libvirt 8.6.0-0ubuntu1
apparmor 3.0.7-1ubuntu1

Creating a VM with virt-install produces this AppAmore denial:

AVC apparmor="DENIED" operation="open" profile="libvirt-974c9859-e682-4f5d-b0cb-dcf3d60185fc" name="/sys/devices/system/cpu/possible" pid=2522 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=64055 ouid=0

Creation of the VM is successful. This is with nested virtualization.

This did not happen with libvirt 8.0.0-1ubuntu8 and apparmor 3.0.7-1ubuntu1.

[Lena Voytek (lvoytek)]

Hello,
Thank you for the bug report. I managed to reproduce the apparmor denial once by creating an Ubuntu Kinetic vm with virt-manager, and then running the following commands:

# sudo apt update && sudo apt dist-upgrade -y
# sudo apt install apparmor virtinst wget -y
# wget https://releases.ubuntu.com/22.04.1/ubuntu-22.04.1-desktop-amd64.iso
# virt-install --osinfo ubuntu-lts-latest -c ubuntu-22.04.1-desktop-amd64.iso --disk size=10 --memory 2048
# sudo dmesg | grep "apparmor=\"DENIED\""

However, every additional time I attempted to reproduce the denial I was unable to, even when creating a new base virtual machine. Does the denial appear for you every time?

If you could also add the specific virt-install parameters to this bug report too that may help.

Thanks!

[Marius Vollmer (marius-vollmer-gmail)]

These are the steps that trigger the denial in our tests (links to images below):

# virt-install --name subVmTest1 --os-variant cirros0.4.0 --boot hd,network --vcpus 1 --memory 128 --import --disk /var/lib/libvirt/images/subVmTest1-2.img --graphics spice,listen=127.0.0.1 --console file,target.type=serial,source.path=/var/log/libvirt/console-subVmTest1.log --print-step 1 > /tmp/xml-system
# virsh define /tmp/xml-system
# virsh start subVmTest1
# virsh shutdown subVmTest1
# virsh start subVmTest1

Each start of subVmTest1 will produce one more DENIED message in the journal, so the above will produce two.

The /var/lib/libvirt/images/subVmTest1-2.img file can be downloaded here (18 MB):

https://cockpit-images.eu-central-1.linodeobjects.com/cirros-ff4ccf16a162d7d3bf86d30141bd8cfe30821dd3b09712fe2f84d201c8e948af.qcow2

The base Ubuntu Kinetic image we use is linked below (1.2 GB), but I would have to dig deeper for the details of how we actually run it. I hope those don't matter...

https://cockpit-images.eu-central-1.linodeobjects.com/ubuntu-stable-000fb42579358bdff9f0422234374540c4f71850a53881708cebf7ace07b7d91.qcow2

[Marius Vollmer (marius-vollmer-gmail)]

I have tried your steps from comment #1 and they do produce one DENIED message for each run of virt-install.

I get errors because of wrong permissions, but the DENIED message still appears.

# virt-install --osinfo ubuntu-lts-latest -c ubuntu-22.04.1-desktop-amd64.iso --disk size=2 --memory 2048
Using default --name ubuntu22.04
WARNING /root/ubuntu-22.04.1-desktop-amd64.iso may not be accessible by the hypervisor. You will need to grant the 'libvirt-qemu' user search permissions for the following directories: ['/root']
WARNING /root/ubuntu-22.04.1-desktop-amd64.iso may not be accessible by the hypervisor. You will need to grant the 'libvirt-qemu' user search permissions for the following directories: ['/root']
WARNING Graphics requested but DISPLAY is not set. Not running virt-viewer.
WARNING No console to launch for the guest, defaulting to --wait -1

Starting install...
Allocating 'ubuntu22.04.qcow2' | 0 B 00:00:00 ...
Removing disk 'ubuntu22.04.qcow2' | 0 B 00:00:00
ERROR internal error: process exited while connecting to monitor: 2022-09-12T10:02:09.421264Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/root/ubuntu-22.04.1-desktop-amd64.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/root/ubuntu-22.04.1-desktop-amd64.iso': Permission denied
Domain installation does not appear to have been successful.
If it was, you can restart your domain by running:
  virsh --connect qemu:///system start ubuntu22.04
otherwise, please restart your installation.

[Marius Vollmer (marius-vollmer-gmail)]

Just to avoid misunderstanding: All VM creations in our tests are successful; the DENIED message seems to be entirely harmless.

[Marius Vollmer (marius-vollmer-gmail)]

We observe this now also in debian-testing with libvirt-daemon 8.5.0-1+b2 and apparmor
3.0.7-1+b2

[Christian Ehrhardt  (paelzer)]

These kind of reads are usually harmless usually (non fatal if denied, but also not a security risk to allow).
You can see in the deny that it is qemu (or libs) triggering it which matches past occasions.
Quite often (we have seen similar by libudev and libusb) they are due to qemu or one of the libs being updated - more likely than libvirt which carries the apparmor rules.

@Lena - from the type of denial I wonder why it would only occur once. Messages from libvirt tend to appear once (as the daemon is persistent) but this isn't one of those.

Steps from here:
1. repo the case (thanks Lena for already doing so)
2. quick bisect which PKG between Jammy/Kinetic causes this
3. use that explanation for a change to the apparmor rules in libvirt upstream
4. backport for SRUs to have the people be less annoyed by that message

[Christian Ehrhardt  (paelzer)]

I was trying the steps from Lena (comment #1) and Marius (comment #2).
But I'm not seeing it. Neither in a container (with kvm allowed for the guest) nor in a nested VM.

After being puzzled for a few seconds I realized that a lib would only ever read /sys/devices/system/cpu/possible on a multi-cpu system and my VM was single CPU.
Changing it to 3 vcpus made it trigger.

And I can see it every time I try to start the guest.

[Christian Ehrhardt  (paelzer)]

This will construct a profile based on the template used by libvirt to test this outside of other elements (using qmeu just booting a kernel image, not more):

$ sudo cp /etc/apparmor.d/libvirt/libvirt-<one that you have> /etc/apparmor.d/test
$ cat /etc/apparmor.d/abstractions/libvirt-qemu | sudo tee -a /etc/apparmor.d/test
$ sudo vim /etc/apparmor.d/test
  Change header to:
profile test flags=(attach_disconnected) {
  #include <abstractions/base>
...
$ echo "/boot/vmlinuz* r," | sudo tee -a /etc/apparmor.d/test
/boot/vmlinuz* r,
$ echo "}" | sudo tee -a /etc/apparmor.d/test
$ sudo systemctl reload apparmor

We can run qemu now directly in that profile and see the problem:

$ sudo aa-exec -p test -- /usr/bin/qemu-system-x86_64 -machine pc-i440fx-kinetic -accel kvm -cpu host -kernel /boot/vmlinuz -nographic -curses

Triggers:
[ 6861.854970] audit: type=1400 audit(1668435695.650:190): apparmor="DENIED" operation="open" class="file" profile="test" name="/sys/devices/system/cpu/possible" pid=2104 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Christian Ehrhardt  (paelzer)]

Once you are happy with the above test we can change the header to confine any qemu:

like:
/usr/bin/qemu-system-x86_64 flags=(attach_disconnected) {
$ sudo systemctl reload apparmor

That will confine qemu without aa-exec.

And with this commandline it does not mess up the console.
/usr/bin/qemu-system-x86_64 -machine pc-i440fx-kinetic -accel kvm -cpu host -kernel /boot/vmlinuz -nographic -serial file:myfile

$ gdb /usr/bin/qemu-system-x86_64
(gdb) handle SIGUSR1 pass nostop noprint
(gdb) set detach-on-fork off
(gdb) catch syscall openat
(gdb) run -machine pc-i440fx-kinetic -accel kvm -cpu host -kernel /boot/vmlinuz -nographic -serial file:myfile

This could have been done easier, but I was afraid (from past lessons learned) that it would elude me. I've found that it is the new libnuma that triggers this.

(gdb) bt
#0 __GI___open64_nocancel (file=file@entry=0x7feea05bd948 "/sys/devices/system/cpu/possible", oflag=oflag@entry=524288) at ../sysdeps/unix/sysv/linux/open64_nocancel.c:39
#1 0x00007feea0519a80 in read_sysfs_file (fname=fname@entry=0x7feea05bd948 "/sys/devices/system/cpu/possible") at ../sysdeps/unix/sysv/linux/getsysstats.c:148
#2 0x00007feea0519ee4 in __GI___get_nprocs_conf () at ../sysdeps/unix/sysv/linux/getsysstats.c:231
#3 0x00007feea04e5862 in posix_sysconf (name=<optimized out>) at ../sysdeps/posix/sysconf.c:626
#4 linux_sysconf (name=<optimized out>) at ../sysdeps/unix/sysv/linux/x86/../sysconf.c:121
#5 __GI___sysconf (name=<optimized out>) at ../sysdeps/unix/sysv/linux/x86/sysconf.c:36
#6 0x00007feea0eb5bdc in ?? () from /lib/x86_64-linux-gnu/libnuma.so.1
#7 0x00007feea12defbe in call_init (l=<optimized out>, argc=argc@entry=12, argv=argv@entry=0x7ffe12cbbec8, env=env@entry=0x7ffe12cbbf30) at ./elf/dl-init.c:70

[Christian Ehrhardt  (paelzer)]

Interesting - that is on the same version of libnuma:
 libnuma1 | 2.0.14-3ubuntu2 | jammy | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x
 libnuma1 | 2.0.14-3ubuntu2 | kinetic | amd64, arm64, armhf, i386, ppc64el, riscv64, s390x

But it really is numa, maybe it is broken on jammy as well but no one has seen it yet.
The following is enough to show it:

$ cat > testnuma.c << EOF
#include <numa.h>
#include <stdio.h>

int main(void)
{
    if(numa_available() < 0){
        printf("No Numa\n");
        return 1;
    }
    return 0;
}
EOF

$ gcc testnuma.c -o testnuma -lnuma

# run under the profile we created above
aa-exec -p /usr/bin/qemu-system-x86_64 ./testnuma

Shows:
[10623.092247] audit: type=1400 audit(1668439456.911:265): apparmor="DENIED" operation="open" class="file" profile="/usr/bin/qemu-system-x86_64" name="/sys/devices/system/cpu/possible" pid=3625 comm="testnuma" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

[Christian Ehrhardt  (paelzer)]

Full trace thorugh libnuma with symbols:

(gdb) bt
#0 __GI___open64_nocancel (file=file@entry=0x7f6833fbd948 "/sys/devices/system/cpu/possible", oflag=oflag@entry=524288) at ../sysdeps/unix/sysv/linux/open64_nocancel.c:39
#1 0x00007f6833f19a80 in read_sysfs_file (fname=fname@entry=0x7f6833fbd948 "/sys/devices/system/cpu/possible") at ../sysdeps/unix/sysv/linux/getsysstats.c:148
#2 0x00007f6833f19ee4 in __GI___get_nprocs_conf () at ../sysdeps/unix/sysv/linux/getsysstats.c:231
#3 0x00007f6833ee5862 in posix_sysconf (name=<optimized out>) at ../sysdeps/posix/sysconf.c:626
#4 linux_sysconf (name=<optimized out>) at ../sysdeps/unix/sysv/linux/x86/../sysconf.c:121
#5 __GI___sysconf (name=<optimized out>) at ../sysdeps/unix/sysv/linux/x86/sysconf.c:36
#6 0x00007f68340f8fb0 in set_configured_cpus () at libnuma.c:587
#7 0x00007f68340f9001 in set_sizes () at libnuma.c:602
#8 0x00007f68340f7d41 in numa_init () at libnuma.c:100
#9 0x00007f6834108f7a in call_init (l=0x7f68341021f0, argc=argc@entry=1, argv=argv@entry=0x7ffd9ca0c4c8, env=env@entry=0x7ffd9ca0c4d8) at ./elf/dl-init.c:56
#10 0x00007f68341090a8 in call_init (env=0x7ffd9ca0c4d8, argv=0x7ffd9ca0c4c8, argc=1, l=<optimized out>) at ./elf/dl-init.c:33
#11 _dl_init (main_map=0x7f683413b2e0, argc=1, argv=0x7ffd9ca0c4c8, env=0x7ffd9ca0c4d8) at ./elf/dl-init.c:117
#12 0x00007f68341218b0 in _dl_start_user () from /lib64/ld-linux-x86-64.so.2

We can check that code, so it is actually:
  sysconf(_SC_NPROCESSORS_CONF)

[Christian Ehrhardt  (paelzer)]

Confirmed:

$ cat testsysconf.c
#include <unistd.h>
#include <stdio.h>

int main(void)
{
    printf("_SC_NPROCESSORS_CONF %ld\n", sysconf(_SC_NPROCESSORS_CONF));
}

$ gcc -Wall testsysconf.c -o testsysconf

$ strace -e openat ./testsysconf
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/sys/devices/system/cpu/possible", O_RDONLY|O_CLOEXEC) = 3
_SC_NPROCESSORS_CONF 3
+++ exited with 0 +++

[Christian Ehrhardt  (paelzer)]

Jammy:

root@j:~# strace -e openat ./testsysconf
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/sys/devices/system/cpu", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
_SC_NPROCESSORS_CONF 4
+++ exited with 0 +++

[Christian Ehrhardt  (paelzer)]

That is the commit causing the change [1] in behavior.

That is pretty low level (in libc6) and will probably hit anything that links against libnuma.

I think the fix should therefore go into
  /etc/apparmor.d/abstractions/base

Today it has:
  # glibc's sysconf(3) routine to determine free memory, etc
  @{PROC}/meminfo r,
  @{PROC}/stat r,
  @{PROC}/cpuinfo r,
  @{sys}/devices/system/cpu/ r,
  @{sys}/devices/system/cpu/online r,

And due to [1] I think this needs to get:
  @{sys}/devices/system/cpu/possible r,

That is still missing in upstreams [2] current base profile.

Gladly it isn't too fatal, but still bad.
Retargetting this to the apparmor package.

[1]: https://sourceware.org/git/?p=glibc.git;a=commit;h=97a912f7a832a6
[2]: https://gitlab.com/apparmor/apparmor/-/blob/master/profiles/apparmor.d/abstractions/base#L98

[Christian Ehrhardt  (paelzer)]

Reported upstream at https://gitlab.com/apparmor/apparmor/-/issues/283

[Christian Ehrhardt  (paelzer)]

Submitted upstream:
 https://lists.ubuntu.com/archives/apparmor/2022-November/012528.html

Once discussed and accepted there I suggest a backport to Kinetic.

I hope this debug and patch helps, but to manage expectations, I'd hope/expect that someone usually looking after apparmor does that follow on step then. Could someone please agree to take it over from here and comment on this bug?

P.S. I mostly want to avoid stepping on someones toes, if you want me to upload it to kinetic I can do so, let me know.

[John Johansen (jjohansen)]

This has now landed upstream, on the master branch as

c159d0925 Allow access to possible cpus for glibc-2.36

and has been cherry-picked back to 3.1, 3.0, 2.13, and 2.12 branches. This schedules it for release in the 2.12.4 and 2.13.7, 3.0.8 releases this week. Unfortunately 3.1.2 was cut last week so it just misses that release.

I will look at assigning resources to this, I am not sure which of us will pick it up yet.

[Marius Vollmer (marius-vollmer-gmail)]

Thanks a lot everyone!

What actually is the effect of the denial? Will qemu not use more than one CPU, or is it something less harmful?

[Christian Ehrhardt  (paelzer)]

Hi Marius,

> What actually is the effect of the denial? Will qemu not use more than one CPU,
> or is it something less harmful?

Since the new interface is arch specific and new the code does fall back tot he old way.

  226 /* On some architectures it is possible to distinguish between configured
  227 and active cpus. */
  228 int
  229 __get_nprocs_conf (void)
  230 {
  231 int result = read_sysfs_file ("/sys/devices/system/cpu/possible");
  232 if (result != 0)
  233 return result;
  234
  235 /* Fall back to /proc/stat and sched_getaffinity. */
  236 return get_nprocs_fallback ();
  237 }

Due to that, even when denied it gets the right number (as it had before).

Once with and without isolation blocking access.

ubuntu@k2:/tmp$ ./testsysconf
_SC_NPROCESSORS_CONF 3

ubuntu@k2:/tmp$ sudo aa-exec -p test -- ./testsysconf
_SC_NPROCESSORS_CONF 3

It only has a real difference on systems where the new code was needed in the first place.
Those are usually rather massive systems which start at lower cpu counts but might hot-plug them later - on those with the denial falling back you'd only get a lower than the real potential max number.
The code that hits this in your case is libnuma on initialization, unless you are very deep into numa control on very huge systems using cpu hotplug you won't see any effect.

[Utkarsh Gupta (utkarsh)]

Ubuntu 22.10 (Kinetic Kudu) has reached end of life, so this bug will not be fixed for that specific release.

[Martin Pitt (pitti)]

Similar issue: https://gitlab.com/libvirt/libvirt/-/issues/548 . These two may want a common fix with "allow qemu to read sysfs"?