Apparmor prevents using storage pools and hostdev networks

Bug #1677398 reported by Simon Déziel on 2017-03-29
36
This bug affects 8 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Medium
Christian Ehrhardt 
Xenial
Undecided
Unassigned
Yakkety
Undecided
Unassigned
Zesty
Undecided
Unassigned

Bug Description

Apparmor prevents qemu-kvm guests from using ZFS volumes.

[Impact]
* storage pools are not usable.
  Examples with zfs and LVM pools

[Test Case 1]
# Prep ZFS
1) Create a zpool
 $ for i in $(seq 1 3); do dd if=/dev/zero of=/tmp/fdisk${i} bs=1M count=1024; done
 $ sudo zpool create internal /tmp/fdisk*
2) Create a ZFS storage pool and volume (named like your zpool, "internal" here)
  $ virsh pool-define-as internal zfs
  $ virsh pool-start internal
  $ virsh vol-create-as internal foo 2G

# prep LVM
4) prepare a (fake) LVM
  $ for i in $(seq 1 3); do dd if=/dev/zero of=/tmp/lvdisk${i} bs=1M count=1024; done
  $ sync
  $ DISKS=$(for i in $(seq 1 3); do sudo losetup -f show /tmp/lvdisk${i}; done)
  $ sudo pvcreate --verbose $DISKS
  $ sudo vgcreate --verbose testvg $DISKS
5) Create LVM Pool and volume
 $ virsh pool-define-as testvg logical
 $ virsh pool-start testvg
 $ virsh vol-create-as testvg guest1 2G

# Prep Guest and use Pools
6) Create a KVM guest e.g. via uvtool
 $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
 $ ssh-keygen
 $ uvt-kvm create --password=ubuntu testguest release=xenial arch=amd64 label=daily
7) Edit the guest's XML profile to use the ZFS and LVM volumes (zvol)
    <disk type='volume' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source pool='internal' volume='foo'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <disk type='volume' device='disk'>
      <driver name='qemu' type='raw'/>
      <source pool='testvg' volume='guest1'/>
      <target dev='vda' bus='virtio'/>
    </disk>
8) Start the guest

The guest refuses to start:

  # virsh start nms
  error: Failed to start domain foo
  error: internal error: process exited while connecting to monitor: 2017-03-29T22:07:31.507017Z qemu-system-x86_64: -drive file=/dev/zvol/internal/foo,format=raw,if=none,id=drive-virtio-disk0,cache=none: Could not open '/dev/zvol/internal/foo': Permission denied

dmesg reveals the culprit:

apparmor="DENIED" operation="open" profile="libvirt-988a8c25-5190-4762-8170-55dc75fc66ca" name="/dev/zd224" pid=23052 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=109 ouid=109
apparmor="DENIED" operation="open" profile="libvirt-988a8c25-5190-4762-8170-55dc75fc66ca" name="/dev/zd224" pid=23052 comm="qemu-system-x86" requested_mask="wr" denied_mask="wr" fsuid=109 ouid=109

Checking /etc/apparmor.d/libvirt/libvirt-$UUID.files shows that no "/dev/zdXX" has been added.

[Additional info]

# lsb_release -rd
Description: Ubuntu 16.04.2 LTS
Release: 16.04

# apt-cache policy libvirt-bin apparmor linux-image-generic
libvirt-bin:
  Installed: 1.3.1-1ubuntu10.8
  Candidate: 1.3.1-1ubuntu10.8
  Version table:
 *** 1.3.1-1ubuntu10.8 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.3.1-1ubuntu10 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
apparmor:
  Installed: 2.10.95-0ubuntu2.5
  Candidate: 2.10.95-0ubuntu2.5
  Version table:
 *** 2.10.95-0ubuntu2.5 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.10.95-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
linux-image-generic:
  Installed: 4.4.0.70.76
  Candidate: 4.4.0.70.76
  Version table:
 *** 4.4.0.70.76 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
        100 /var/lib/dpkg/status
     4.4.0.21.22 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: libvirt-bin 1.3.1-1ubuntu10.8
ProcVersionSignature: Ubuntu 4.4.0-70.91-generic 4.4.49
Uname: Linux 4.4.0-70-generic x86_64
NonfreeKernelModules: zfs zunicode zcommon znvpair zavl
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
Date: Wed Mar 29 17:48:06 2017
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.default.libvirt-guests: [modified]
modified.conffile..etc.libvirt.qemu.conf: [modified]
modified.conffile..etc.libvirt.qemu.networks.default.xml: [modified]
mtime.conffile..etc.default.libvirt-guests: 2016-08-29T21:09:57.632048
mtime.conffile..etc.libvirt.qemu.conf: 2017-03-29T17:26:03.924234
mtime.conffile..etc.libvirt.qemu.networks.default.xml: 2016-04-23T19:24:13.505208

Simon Déziel (sdeziel) wrote :

Hi Simon,
thanks for your report - so we did not get far enough with bug 1641618 which only solved things for direct zvols, but not for disks from a pool.

I'm afraid there might be no part generating that yet in the aa-helper, but I'll look into it and report back here once I know more details.

Extending your already good testcase description:

# create a simple guest
 $ sudo apt-get install uvtool-libvirt zfsutils-linux
 $ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
 $ ssh-keygen
 $ uvt-kvm create --password=ubuntu testguest release=xenial arch=amd64 label=daily
# create a zpool to use
 $ for i in $(seq 1 3); do dd if=/dev/zero of=/tmp/fdisk${i} bs=1M count=1024; done
 $ sudo zpool create internal /tmp/fdisk*
# make pool in libvirt and guest disk foo
 $ virsh pool-define-as internal zfs
 $ virsh pool-start internal
 $ virsh vol-create-as internal foo 2G
# link up zpool, by adding this to the guest
    <disk type='volume' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source pool='internal' volume='foo'/>
      <target dev='vdc' bus='virtio'/>
    </disk>
# start the guest
$ virsh start testguest

All run into:
Could not open '/dev/zvol/internal/foo': Permission denied

And I can see the reported Deny:
apparmor="DENIED" operation="open" [...] name="/dev/zd0" [...]

That said setting to confirmed for now.
Also I checked this applies to all of releases X-Z.

Need to dive into aa-helper how close or far that is as of today to get this done.

Changed in libvirt (Ubuntu):
status: New → Confirmed
Changed in libvirt (Ubuntu Yakkety):
status: New → Confirmed
Changed in libvirt (Ubuntu Xenial):
status: New → Confirmed

These are mostly notes to remember later:
In such a setup the pool definition has the base pool path

$ virsh pool-dumpxml internal
<pool type='zfs'>
  <name>internal</name>
  <uuid>5e83970c-dc95-41af-bd10-9d9001dc9cba</uuid>
  <capacity unit='bytes'>3170893824</capacity>
  <allocation unit='bytes'>2147573760</allocation>
  <available unit='bytes'>1023320064</available>
  <source>
    <name>internal</name>
  </source>
  <target>
    <path>/dev/zvol/internal</path>
  </target>
</pool>

The volume holds the respective subvol foo

$ virsh vol-dumpxml --pool internal foo
<volume type='block'>
  <name>foo</name>
  <key>/dev/zvol/internal/foo</key>
  <source>
  </source>
  <capacity unit='bytes'>2147483648</capacity>
  <allocation unit='bytes'>2147483648</allocation>
  <target>
    <path>/dev/zvol/internal/foo</path>
  </target>
</volume>

And confimring as well that /etc/apparmor.d/libvirt/libvirt-<UUID>.files has no references to either /dev/zvol/internal/foo nor the symlink target.

Apparmor can't by design follow symlinks (https://bugs.launchpad.net/apparmor/+bug/1485055).
So test-inserting into /etc/apparmor.d/abstractions/libvirt-qemu:
- /dev/zvol/internal/foo rw, => still fails
- /dev/zd0 rw, => works (guest sees disk as expected)
So does any generic rule.

So the following might serve as a temporary workaround adding "/dev/zd[0-9]* rw" to /etc/apparmor.d/abstractions/libvirt-qemu.
Simon I'm sure you had that already, but this is for whoever else comes by.

I see that this needs dev-activity -> upstream-libvirt -> merge new libvirt -> SRUs so I wanted to provide some sort of workaround.

TODO:
- get aa-helper to consider pool zvols
- resolve symlink as we need the target in the rule

Hello Christian,

On 2017-03-30 06:18 AM, ChristianEhrhardt wrote:
> So the following might serve as a temporary workaround adding "/dev/zd[0-9]* rw" to /etc/apparmor.d/abstractions/libvirt-qemu.

What I did something similar but less convenient. My goal was to keep
the per-VM isolation so I added the corresponding "/dev/zdXX rw" rule to
the /etc/apparmor.d/libvirt/libvirt-$uuid file and reload that profile.

> I see that this needs dev-activity -> upstream-libvirt -> merge new
> libvirt -> SRUs so I wanted to provide some sort of workaround.

Yes, makes sense and your workaround is easier. Having this eventually
land in a SRU would be greatly appreciated.

> TODO:
> - get aa-helper to consider pool zvols
> - resolve symlink as we need the target in the rule

That is correct, Apparmor always operate on the destination file. There
should already be code in aa-helper to track down the destination file
as I assume the situation is pretty similar to that of LVM.

As always, thanks for the precise problem dissection and fast response!

FYI - I haven't forgotten, just flooded with other load

Damn it seems I can't find the hours yet - I really beg your pardon Simon as I like you as an active community member! But I also have PTO next week and I'm not sure I get to it before that.
I'll assign to myself to not forget about it.

Changed in libvirt (Ubuntu Zesty):
assignee: nobody → ChristianEhrhardt (paelzer)
tags: added: server-next

No worries, I have a good feeling of how busy you are from the bug
notifications I get. Knowing that you will look into it is already a
great deal, so thanks again.

On 2017-04-05 03:33 AM, ChristianEhrhardt wrote:
> Damn it seems I can't find the hours yet - I really beg your pardon Simon as I like you as an active community member! But I also have PTO next week and I'm not sure I get to it before that.
> I'll assign to myself to not forget about it.
>
> ** Changed in: libvirt (Ubuntu Zesty)
> Assignee: (unassigned) => ChristianEhrhardt (paelzer)
>
> ** Tags added: server-next
>

tags: added: virt-aa-helper
tags: removed: server-next

I don't seem to find time shortly, so I drop server-next tag.
But I ensured it is tracked to remind me and make me feel bad :-/

Changed in libvirt (Ubuntu Yakkety):
status: Confirmed → Won't Fix
Changed in libvirt (Ubuntu):
status: Confirmed → In Progress
Changed in libvirt (Ubuntu Zesty):
assignee: ChristianEhrhardt (paelzer) → nobody

Back on this, currently trying to build up a case where this can be tested from git (had some obstacles):
- In the dev system (from local dir) not all apparmor rules apply
- In a container the zfs actions are not all possible
- So we need a KVM driving a 2nd-level KVM for all of this.

0. get a multi-cpu KVM guest with build env

1. normal uvtool based guest in there
2. prep zfs as outlined in c#3
4. check if bug triggers and confinement is active
   $ sudo aa-status | grep -E 'libv|qemu'
5. share the repo dir
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/home/paelzer/work/libvirt/libvirt-upstream-git-root'/>
      <target dir='libvirt-git'/>
    </filesystem>
   And then in guest:
   $ sudo mkdir -p /home/paelzer/work/libvirt/libvirt-upstream-git-root
   $ sudo mount -t 9p -o trans=virtio libvirt-git /home/paelzer/work/libvirt/libvirt-upstream-git-root
5. switch to locally built repo
   (built on host and used in guest as root), install into the system
   $ sudo make install
6. check you have the new version
7. Check contained aa status
   $ sudo aa-status | grep -E 'libv|qemu'
8. check the bug still triggers running from that

That is close to a ppa build and install being easier :-)
It is also easier to retest for others on the bug and more reliable to catch the way will work in Ubuntu.
So while (somehwat) working gogin on with local dev and then shoving it onto test systems through a ppa build.

Initially this can be run locally just with virt-aa-helper as any virt-aa-helper dev.
This mostly is like c#3, but with a service from local build and with libvtool wrapper to gdb the virt-aa-helper.

Since even this is not the most straight forward thing if you never done it here a short log how to do so:

 $ sudo ./src/virtlockd -f /etc/libvirt/virtlockd.conf -d
 $ sudo ./src/virtlogd -f /etc/libvirt/virtlogd.conf -d
 $ sudo ./daemon/libvirtd -f /etc/libvirt/libvirtd.conf -d
 # an xml containing the snippet of c#3
    <disk type='volume' device='disk'>
      <driver name='qemu' type='raw' cache='none'/>
      <source pool='internal' volume='foo'/>
      <target dev='vdc' bus='virtio'/>
    </disk>
 $ ./tools/virsh (all you need to set up the pool as in c#3)
 # run like:
 $ ./src/virt-aa-helper --lt-debug --create --dryrun --uuid 'libvirt-e2f807e2-4ee6-496f-b3cc-926b7c7cefb3' < ../bug-1677398-zfs-pool/test1.xml
 # debug like:
 $ libtool --mode=execute gdb ./src/virt-aa-helper
 (gdb) run --create --dryrun --uuid 'libvirt-e2f807e2-4ee6-496f-b3cc-926b7c7cefb3' < ../bug-1677398-zfs-pool/test1.xml

As we knew the arg to qemu uses the link like:
  -drive file=/dev/zvol/internal/foo,format=raw
But we need the base device, so in virt-aa-helper we need to:
1. the volume is a disk entry (not an FS), so we need to get the volume entry from the guest xml
2. from pool+volume get to the real path (like the -drive arg construction would do)
3. need to readlink to the link target.

When the guest gets parsed all that virt-aa-helper initially has is the disk element.
This has in our case only:
p *(disk->src->srcpool)
$10 = {pool = 0x100236b90 "internal", volume = 0x100236440 "foo", voltype = 0, pooltype = 0, actualtype = 0, mode = 0}

Nothing in virt-aa-helper yet cares about that - it does not deliver a "virDomainDiskGetSource" and thereby gets skipped.
But with that we would know pool and volume (step #1 above).

Next is getting the pool details from that, to do so we align to what the actual usage of the attribute does.
In virStorageTranslateDiskSourcePool is that flow which mostly is
- virStoragePoolLookupByName
  -> virStoragePoolIsActive (check if active)
  -> virStorageVolLookupByName (now one has the volume)
  -> virStorageVolGetPath

It seems designed to be safe to use virStorageTranslateDiskSourcePool, the overall pattern is to call it before handling the other attributes. It's return value is 0 if it was no srcpool disk anyway. If it is a sourcepool but it failed for any reason then it is -1.
That makes it a safe pre-call before running on the disk elements data like def->src->path whicih it sets.

That structurally seems to be what all other code does, so it might be wise to follow that.
The problem is that this was meant to be called with an active connection (first argument).
This can not be avoided by calling the sub-functions directly as this is needed in virStoragePoolLookupByName as the storage drivers it calls are in that struct.

virConnectOpenReadOnly and co would initialize the connection, but they would be env specific.
Also as it seems the header is meant for users of libvirt, but not the local tools.
I'll report back when I found the proper initialization (e.g. the empty one via virGetConnect works but is obviously uninitialized and rejects the lookup method).

I'm not sure if upstream likes that virt-aa-helper now needs to talk to the socket, but they can suggest otherwise if they don't.
I now have something that calls the lookup correctly.
It then fails to find the pool:
  libvirt: Storage Driver error : Storage pool not found: no storage pool with matching name 'internal'

While at the sametime
$ ./tools/virsh pool-info internal
Name: internal
UUID: 9bdda23f-3410-4a2c-9f93-850308d19445
State: running

Going on here tmrw

After having slept on that for a night I consider that not a good way.
The current lifecycle of virt-aa-helper just doesn't match to create a local virConnectPtr context, yet OTOH without one we can't translate the pool as the code that spawns the guest would.

Before I implement some useless code to be nack'ed for its misconception I readed out to the libvirt devel mailing list outlining these issues and hope that feedback from there unblocks me.

Going on with another virt-aa-helper bug until the discussion resolved.

[1]: https://www.redhat.com/archives/libvir-list/2017-September/msg00557.html

Changed in libvirt (Ubuntu):
status: In Progress → Triaged
summary: - Apparmor prevents using ZFS storage pools
+ Apparmor prevents using storage pools
Changed in libvirt (Ubuntu):
importance: Undecided → Medium

I found that bug 1343245 is about the same general issue.
The descriptions in there were great, but since I started to document the debug and potential coding and more here I dupp'ed it onto here.
The thoughts there also already were around "how to get the translation done".

I changed title and description here - especially of the testcase - to now cover both.
This shall show that it is a more general issue.

On the bug itself still waiting for upstream feedback to my mail linked in c#16

description: updated

In the same scope of required "out of context information" fall cases of vfio devices for hostdevs.
Those work fine if defined in the guest or added to the guest.

But if only referred by an interface like:
    <interface type='network'>
      <mac address='3a:73:83:14:99:0e'/>
      <source network='pf-et0p0' portgroup='storage'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>

And the definition being external like:
<network>
  <name>pf-et0p0</name>
  <uuid>c1415c6d-11d7-417e-8113-ed5439e5ee44</uuid>
  <forward mode='hostdev' managed='yes'>
    <driver name='vfio'/>
    <pf dev='et0p0'/>
  </forward>
  <portgroup name='ext' default='yes'>
    <vlan>
      <tag id='30'/>
    </vlan>
  </portgroup>
  <portgroup name='lab2'>
    <vlan>
      <tag id='51'/>
    </vlan>
  </portgroup>
  <portgroup name='storage'>
    <vlan>
      <tag id='61'/>
    </vlan>
  </portgroup>
</network>

It fails.
As again in this case virt-aa-helper has no means yet to introspect the extra info needed to convert all this to paths.

summary: - Apparmor prevents using storage pools
+ Apparmor prevents using storage pools and hostdev networks

TODO: retest these with the domain label callbacks implemented, maybe some of the devices/images might trigger that

tags: added: libvirt-apparmor-dev
ultrabit (ultrabit) wrote :

I have the same problem using raw lvm logical volumes as disk on Ubuntu 18.04.
When i try to start a vm with virt-manager qemu says Permission denied on device.

The lvm uses device mapper to map the logical volumes so i need to handle devices like

brw-rw---- 1 libvirt-qemu kvm 253, 4 mar 17 13:24 /dev/dm-4

After a bunch of failure attempts to start the vm machine, the following steps was successful:

I run the aa-logprof (apparmor-utils package) util to update the apparmor profiles using the syslog, and
i add the following line in /etc/apparmor.d/abstractions/libvirt-qemu

...
  /dev/dm* krw,

just before the /dev/net/tun rw, line.

If the line already exists just replace "rw" with "krw" to permit file lock operations.

Now it seems vm runs well again.

Seth Arnold (seth-arnold) wrote :

Hi Ultrabit, can you please include the DENIED lines from your dmesg or auditd logs?

Thanks

Up, causing issues with terraform libvirt provider :/

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers