[xenial] No write access to VirtFS (9p) in qemu VM run by libvirt

[Harald Hetzner]

Using virt-manager and libvirtd, I added a VirtFS filesystem passthrough to an existing qemu virtual machine also running Ubuntu.

The XML code generated by libvirt looks like this:

    <filesystem type='mount' accessmode='mapped'>
      <source dir='/p9test'/>
      <target dir='p9_mapped'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </filesystem>

Inside the VM, I am able to mount the filesystem passthrough like this:

mount -t 9p -o trans=virtio,version=9p2000.L,rw p9_mapped /mnt

After mounting, it is possible to read the contents of the passthrough, but it is not possible to write into the directory. E.g. 'touch /mnt/test' results in the error:

touch: cannot touch ‘/mnt/test’: Permission denied

Using the other p9 access modes 'passthrough' and 'squash' also does not work. Read is possible, write isn't.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: libvirt-bin 1.3.1-1ubuntu6
ProcVersionSignature: Ubuntu 4.4.0-13.29-generic 4.4.5
Uname: Linux 4.4.0-13-generic x86_64
ApportVersion: 2.20-0ubuntu3
Architecture: amd64
CurrentDesktop: Unity
Date: Fri Mar 18 22:12:34 2016
SourcePackage: libvirt
UpgradeStatus: Upgraded to xenial on 2016-02-06 (41 days ago)
modified.conffile..etc.libvirt.qemu.conf: [inaccessible: [Errno 13] Keine Berechtigung: '/etc/libvirt/qemu.conf']
modified.conffile..etc.libvirt.qemu.networks.default.xml: [inaccessible: [Errno 13] Keine Berechtigung: '/etc/libvirt/qemu/networks/default.xml']

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug.

Who is /mnt/test owned by? what do

ls -ld /mnt/test
ls -l /mnt/test

show? Libvirt launches qemu as the libvirt-qemu user, which is
probably not allowed to create files there. If that is the case, then
ou can either change the ownership/permissions of the shared directory
or change the user which qemu runs as, which is specified in
/etc/libvirt/qemu.conf

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[mfg (micahgalizia)]

Hi I have this problem now too (it worked before upgrading to xenial from trusty). My share is /srv/video on the host and video in my home folder on the host. My ls commands output

host:

drwxrwxr-x 7 micah users 4096 Apr 14 20:49 video/

guest:

drwxrwxr-x 7 micah users 4096 Apr 14 20:49 video/

host config:

    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/srv/video'/>
      <target dir='video'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </filesystem>

fstab on the guest:

video /home/micah/video 9p trans=virtio,rw 0 0

and from the host syslog:

Apr 14 20:48:14 mastermold kernel: [14059.033861] audit: type=1400 audit(1460681294.791:23): apparmor="DENIED" operation="chown" profile="libvirt-a3ede2b7-63d4-bcfb-8342-724f20a8cc48" name="/srv/video/" pid=3060 comm="qemu-system-x86" requested_mask="w" denied_mask="w" fsuid=106 ouid=1000

This error has come up before, but there doesn't appear to be a workaround...

[Serge Hallyn (serge-hallyn)]

Hi,

a workaround should be to add

  /srv/video/ w,

to /etc/apparmor.d/abstractions/libvirt-qemu.

For this to have regressed since 14.04 it seems qemu must have started
chowning the file where it didn't before. The correct fix is for
virt-aa-helper to detect these and add an exception when needed.

[Jessie Bryan (jessie-bryan+ubuntu)]

I haven't had any luck using the workaround. Once i update /etc/apparmor.d/abstractions/libvirt-qemu with my updates (/zfs/files/ w,) I then restarted the apparmor service via systemctl. Afterwards, I shutdown my guest VM and booted it up. I then attempted to write files via the 9p virtio mount and failed. I can read files just fine.

FWIW, I even stopped apparmor and still no sucess writing.

[Bogdan Yurov (nick4fake)]

The workaround does not work for me either.

[Bogdan Yurov (nick4fake)]

Running qemu/kvm as root, setting type to passthrough did not help also.

[Christian Ehrhardt  (paelzer)]

Hmm,
it seems this fell through sorry for that.
With the workaround Serge suggested is the apparmor denial still just the same e.g.:
apparmor="DENIED" operation="chown" profile="libvirt-a3ede2b7-63d4-bcfb-8342-724f20a8cc48" name="/srv/video/" pid=3060 comm="qemu-system-x86" requested_mask="w" denied_mask="w" fsuid=106 ouid=1000

[Christian Ehrhardt  (paelzer)]

Adding to the list of virt-aa-helper extensions needed.
Yet I still look for someone to confirm that when the workaround is applied (matching your custom dir) if then it is still apparmor that blocks you (and with which message)?

[sgofferj (sgofferj)]

I see similar problems, however, I don't see any apparmor denys.
Actually, I'm totally puzzled by the effects I'm seeing...

I import a ZFS partition as p9 into a virtual machine which runs a webserver. Everything worked fine under 14.04. Under 16.04 I see the following problems:

No normal user can modify files
Normal users can touch existing files which belong to them
Normal users canNOT create files
Normal users can delete files that belong to them
root can modify files which belong to root
root canNOT modify files which belong to another user

I DID add the respecitve ZFS pertitions to /etc/apparmor.d/local/usr.sbin.libvirtd:
/storage/asterisk/spool/** lrwmk,
/storage/webserver/** lrwmk,,

As mentioned, I do not see any related apparmor denys in the syslog.

I'm running an Owncloud instance on that server which currently is effectively useless so either solving this problem or a workaround would be very much appreciated.

Suggest increasing importance as the bug breaks basic functionality.

[sgofferj (sgofferj)]

It seems, I have found the issue at least for my side. I noticed the following message in the kernel log:

Sep 16 19:14:36 nostromo kernel: [ 8050.077165] audit: type=1400 audit(1505578476.590:111103): apparmor="DENIED" operation="capable" profile="libvirt-ab5c87f8-7085-be26-548e-d9433b84af89" pid=11171 comm="qemu-system-x86" capability=3 capname="fowner"

Further investigation revealed that the update had overwritten my customized /etc/apparmor.d/abstractions/libvirt-qemu.

I added the following statements to the file right at the top after capability chown:
  capability fowner,
  capability fsetid,

That solved all of my access issues.

[Christian Ehrhardt  (paelzer)]

Ok, so we need three things:
1. the zfs rules to be generated which is bug 1677398
2. for this one here understand the video rule if/how it is releated and generate it accordingly
3. check where/why qemu does these fowner/fsetid things and create a rule for it depending on that.
   If it does so in general (and for now as a workaround) add it to the abstractions/libvirt-qemu
   but if we can track down to e.g. only 9p then we should generate that into the per-guest rules.

Thanks a lot sgofferj for the update!

I'm currently working on a set of apparmor issues related to virt-aa-helper, this takes some time as I debug and dev them one by one, but this should be part of it rather soon.

[Christian Ehrhardt  (paelzer)]

I wasn't awake this morning it seems (or did to omuch at once), so I beg your pardon and resummarize.

Also I had the chance to try the fs forwarding on a zestyl level libvirt/qemu and it worked fine.

- The /srv/video rule obviously is just the case reported for a share that exports this source.
  That is the actual bug here that a rule for that has to be generated.

On Zesty that seems to work, for a xml entry like the following:
    <filesystem type='mount' accessmode='passthrough'>
      <source dir='/home/paelzer/work/libvirt/libvirt-upstream-git-root'/>
      <target dir='libvirt-git'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </filesystem>
I got generated apparmor rules:
    "/home/paelzer/work/libvirt/libvirt-upstream-git-root/**" rwl,
    "/home/paelzer/work/libvirt/libvirt-upstream-git-root/" r,

And it works with rw all the way (sharing a git tree shared between host and guest).

1. So since this bug is about the rule creation it seems that exists, needs to be identified and backported for Xenial.

2. about the report by sgofferj this morning I wonder as I have no fowner/fsetid denials.
   Maybe this is specific to exports based on zfs.
   @sgofferj - would you mind opening a new bug for this and attach your guest XML as well as a
   description of your ZFS setup there? I want to understand and track down your case, but keep
   it out of this bug here (which is about the source path not added to the rules)