libvirt's apparmor policy prevents starting domain with hugepage-backed memory store

[bugproxy]

---Problem Description---
After enabling hugepages, unable to start a domain with hugepage-packed memory store:

~# virsh start ubuntu-14_04
error: Failed to start domain ubuntu-14_04
error: internal error: process exited while connecting to monitor: 2015-07-10T14:26:04.627101Z qemu-system-ppc64le: unable to create backing store for hugepages: Permission denied

---uname output---
Linux cipipeln 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 19:33:37 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

---Steps to Reproduce---
1. set the sysctl setting for hugepages to 5000
2. enable huge pages in /etc/default/qemu-kvm
3. restart qemu-kvm
4. restart libvirt
5. add to guest xml (using virsh edit)
<memoryBacking>
   <hugepages/>
</memoryBacking>
6. virsh define ubuntu-14_04.xml
7. virsh start ubuntu-14_04

====
# grep Huge /proc/meminfo
AnonHugePages: 0 kB
HugePages_Total: 5000
HugePages_Free: 5000
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 16384 kB

# cat /etc/default/qemu-kvm
# To disable qemu-kvm's page merging feature, set KSM_ENABLED=0 and
# sudo restart qemu-kvm
KSM_ENABLED=1
SLEEP_MILLISECS=200
# To load the vhost_net module, which in some cases can speed up
# network performance, set VHOST_NET_ENABLED to 1.
VHOST_NET_ENABLED=1

# Set this to 1 if you want hugepages to be available to kvm under
# /run/hugepages/kvm
KVM_HUGEPAGES=1

==
File permissions on the hugepages mount dir:

root@cipipeln:~# ls -lah /dev | grep hugep
drwxr-xr-x 3 root root 0 Jun 16 12:39 hugepages
root@cipipeln:~# ls -lah /dev/hugepages/
total 0
drwxr-xr-x 3 root root 0 Jun 16 12:39 .
drwxr-xr-x 18 root root 4.9K Jul 8 11:03 ..
drwxr-xr-x 3 root root 0 Jun 16 12:39 libvirt
root@cipipeln:~# ls -lah /dev/hugepages/libvirt/
total 0
drwxr-xr-x 3 root root 0 Jun 16 12:39 .
drwxr-xr-x 3 root root 0 Jun 16 12:39 ..
drwxr-xr-x 2 libvirt-qemu kvm 0 Jul 9 18:34 qemu

==

# sysctl -a | grep huge
vm.hugepages_treat_as_movable = 0
vm.hugetlb_shm_group = 0
vm.nr_hugepages = 5000
vm.nr_hugepages_mempolicy = 5000
vm.nr_overcommit_hugepages = 0

==

I was able to confirm that running qemu-kvm stand-alone was allocated 4 hugepages (The number of free huge pages decreased by 4).

==
I also tried:
- sysctl -w vm.hugetlb_shm_group=X where X was the user id for libvirt, and then tried the group id for the kvm group
- disabling apparmor
- chmod -R 777 /dev/hugepages

Hi Christy, Are you using apparmor for any specific reason ? If you can switch to selinux, it would be convinient for me to debug as I am very comfortable with selinux. The issue I see is qemu is not able to create a temporary file in /dev/hugepages/libvirt/qemu/qemu_back_mem.ppc_spapr.ram.XXX.

Can you disable that and try Or confirm if the /dev/hugepages directory has relaxed security? I tried relaxing a bit and failed as I dont know apparmor well.

Thanks,
Shiva

== Comment: #8 - Christy L. Norman Perez <email address hidden> - 2016-01-07 11:33:19 ==
(In reply to comment #7)
> Hi Christy, Are you using apparmor for any specific reason ? If you can
> switch to selinux, it would be convinient for me to debug as I am very
> comfortable with selinux. The issue I see is qemu is not able to create a
> temporary file in
> /dev/hugepages/libvirt/qemu/qemu_back_mem.ppc_spapr.ram.XXX.

Nope, not using apparmor for any reason aside from the fact that it's default on Ubuntu.

>
> Can you disable that and try Or confirm if the /dev/hugepages directory has
> relaxed security? I tried relaxing a bit and failed as I dont know apparmor
> well.

I can't believe I didn't look into this before, but it does look like an apparmor issue:
Jan 07 11:17:17 humphrey kernel: audit: type=1400 audit(1452183437.308:281): apparmor="DENIED" operation="mknod" profile="libvirt-be593664-727a-4905-bdb8-2eb7ccf85700

So, here's what I did to get this to work:
root@humphrey:~# aa-complain /etc/apparmor.d/libvirt/libvirt-be593664-727a-4905-bdb8-2eb7ccf85700
Setting /etc/apparmor.d/libvirt/libvirt-be593664-727a-4905-bdb8-2eb7ccf85700 to complain mode.

root@humphrey:~# virsh start bz127508
Domain bz127508 started

>
> Thanks,
> Shiva

I think we should reroute this to apparmor, mirror to Ubuntu, and request an updated apparmor policy. This change should be done for 14.04 and applicable releases after.

Thanks for the suggestion, Shiva!

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

Default Comment by Bridge

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-01-07 22:36 EDT-------
Some clarification is in order. The host was running 15.04 and the client that was to be started (but failed due to the policy on the host) was using 14.04. 14.04 was not used as the host and it is unknown whether or not it has the same issue.

[Seth Arnold (seth-arnold)]

Christy, can you please include the full relevant DENIED lines from your logs so that we can best determine which rules need to be added to the libvirt profiles?

Thanks

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-01-07 23:03 EDT-------
That was the only one, actually.

I just recreated it just to make sure, so these have a later timestamp:

Jan 07 17:55:48 humphrey audit[71908]: AVC apparmor="DENIED" operation="mknod" profile="libvirt-be593664-727a-4905-bdb8-2eb7ccf85700" name="/dev/hugepages/libvirt/qemu/qemu_back_mem.ppc_spapr.ram.Aejg6t" pid=71908 comm="qemu-system-ppc" requested_mask="c" denied_mask="c" fsuid=109 ouid=109
Jan 07 17:55:48 humphrey kernel: audit: type=1400 audit(1452207348.544:338): apparmor="DENIED" operation="mknod" profile="libvirt-be593664-727a-4905-bdb8-2eb7ccf85700" name="/dev/hugepages/libvirt/qemu/qemu_back_mem.ppc_spapr.ram.Aejg6t" pid=71908 comm="qemu-system-ppc" requested_mask="c" denied_mask="c" fsuid=109 ouid=109

[Seth Arnold (seth-arnold)]

See also 1524737.

Thanks

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2016-03-23 07:33 EDT-------
Any updates?

[Seth Arnold (seth-arnold)]

vsandhya, it appears the bugproxy is broken, this is marked as a duplicate of 1524737 which was changed to 'fix released' on 2015-12-28.

Checking the Xenial libvirt package shows that the AppArmor rules have been updated for systemd's new path:
$ grep -r huge debian/apparmor/
debian/apparmor/libvirt-lxc: # allow hugetlbfs mounts everywhere
debian/apparmor/libvirt-lxc: mount fstype=hugetlbfs,
debian/apparmor/libvirt-qemu: # for access to hugepages
debian/apparmor/libvirt-qemu: owner "/run/hugepages/kvm/libvirt/qemu/**" rw,
debian/apparmor/libvirt-qemu: owner "/dev/hugepages/libvirt/qemu/**" rw,

If you're still seeing errors please open a new bug. Thanks.