libvirt with securty_driver="apparmor" (default settings) cannot do live blockcopy of devices due to permission denied error

[Luca Lazzeroni]

Hi,
if I create a VM using default libvirt settings and try to do a blockcopy of one of its block devices, procedure fails claiming "permission denied" and the original block device looses write permission. Only power-cycling the VM restores correct behaviour.

If I manually edit /etc/libvirt/qemu.conf and set

security_driver="none"

then blockcopy works as expected.

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: libvirt-bin 1.1.1-0ubuntu8
ProcVersionSignature: Ubuntu 3.11.0-12.19-generic 3.11.3
Uname: Linux 3.11.0-12-generic x86_64
ApportVersion: 2.12.5-0ubuntu2.1
Architecture: amd64
Date: Wed Nov 6 15:57:33 2013
InstallationDate: Installed on 2013-11-04 (2 days ago)
InstallationMedia: Ubuntu-Server 13.10 "Saucy Salamander" - Release amd64 (20131016)
MarkForUpload: True
SourcePackage: libvirt
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.apparmor.d.abstractions.libvirt.qemu: [modified]
modified.conffile..etc.apparmor.d.local.usr.sbin.libvirtd: [modified]
modified.conffile..etc.libvirt.libvirtd.conf: [modified]
modified.conffile..etc.libvirt.qemu.conf: [modified]
modified.conffile..etc.libvirt.qemu.networks.default.xml: [deleted]
mtime.conffile..etc.apparmor.d.abstractions.libvirt.qemu: 2013-11-06T12:40:14.384226
mtime.conffile..etc.apparmor.d.local.usr.sbin.libvirtd: 2013-11-06T15:02:46.028029
mtime.conffile..etc.libvirt.libvirtd.conf: 2013-11-06T11:17:34.844340
mtime.conffile..etc.libvirt.qemu.conf: 2013-11-06T15:49:54.023964

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug. Could you please show exactly where and
how you are initiating the blockcopy?

 status: incomplete

[Luca Lazzeroni (luca-m)]

Yes, of course.
I developed a live-backup script (I attached it; the script is “bkvm” and you can run it with “-h” to get help) which does the follow:

1) Undefine VM to make it transient
2) Via blockcopy backup all block devices associated to VM
3) Suspend vm
4) Save VM memory state
5) Restore vm to running state (necessary because saving vm-state shutdown the vm)
6) Re-define VM from xml

This script works correctly on ubuntu 13.04 with backported qemu 1.5 - libvirt 1.0.9. I’m using it on 4 servers without problems.

Now I’ve upgraded a server to ubuntu 13.10 (reinstalled it from scratch and moved vm onto it) and the procedure doesn’t work anymore.

By manually executing:

virsh undefine TestVM
virsh blockcopy TestVM vda /mnt/nfs/TestVMBackup/vda-disk.qcow2 —wait —verbose

I got the error. After error (which seems related to a “profile_update” in app armor) I’ve found that in /etc/apparmor.d/libvirt/libvirt-xxxxxx.files the line related to base VM disk disappear, while it appears a new line related to blockcopy target. So apparmor starts to deny access to VM’s disk and vm starts claiming disk-errors.

If I disable security_driver in /etc/libvirt/qemu.conf, however, and restart libvirt-bin, everything works as expected.
I must also add that with apparmor enabled I’ve got host’ syslog flooded by messages denying access to many files.

Manually adding access permissions to target nfs directory in /etc/apparmor.d/local/usr.sbin.libvirtd doesn’t help.

Thank you for your help,

Il giorno 06/nov/2013, alle ore 19:23, Serge Hallyn <email address hidden> ha scritto:

> Thanks for reporting this bug. Could you please show exactly where and
> how you are initiating the blockcopy?
>
> status: incomplete
>
>
> ** Changed in: libvirt (Ubuntu)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1248577
>
> Title:
> libvirt with securty_driver="apparmor" (default settings) cannot do
> live blockcopy of devices due to permission denied error
>
> Status in “libvirt” package in Ubuntu:
> Incomplete
>
> Bug description:
> Hi,
> if I create a VM using default libvirt settings and try to do a blockcopy of one of its block devices, procedure fails claiming "permission denied" and the original block device looses write permission. Only power-cycling the VM restores correct behaviour.
>
> If I manually edit /etc/libvirt/qemu.conf and set
>
> security_driver="none"
>
> then blockcopy works as expected.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 13.10
> Package: libvirt-bin 1.1.1-0ubuntu8
> ProcVersionSignature: Ubuntu 3.11.0-12.19-generic 3.11.3
> Uname: Linux 3.11.0-12-generic x86_64
> ApportVersion: 2.12.5-0ubuntu2.1
> Architecture: amd64
> Date: Wed Nov 6 15:57:33 2013
> InstallationDate: Installed on 2013-11-04 (2 days ago)
> InstallationMedia: Ubuntu-Server 13.10 "Saucy Salamander" - Release amd64 (20131016)
> MarkForUpload: True
> SourcePackage: libvirt
> UpgradeStatus: No upgrade log present (probably fresh install)
> modified.conffile..etc.apparmor.d.abstractions.libvirt.qemu: [modified]
> modified.con...

[Serge Hallyn (serge-hallyn)]

 status: new

[Serge Hallyn (serge-hallyn)]

 status: confirmed

[Serge Hallyn (serge-hallyn)]

Interestingly, /etc/apparmor.d/libvirt/libvirt-$uuid.files does have an
entry allowing rw to the destination path:

  "/mnt/x/x.qcow2" rw,

but still I get

Nov 6 21:29:47 kvm-s1 kernel: [ 1432.501141] type=1400 audit(1383769787.515:60): apparmor="STATUS" operation="profile_replace" parent=1382 profile="unconfined" name="libvirt-5d349701-09af-42be-b1b4-ef4b31de5792" pid=1383 comm="apparmor_parser"
Nov 6 21:29:47 kvm-s1 kernel: [ 1432.502753] type=1400 audit(1383769787.515:61): apparmor="DENIED" operation="open" parent=1 profile="libvirt-5d349701-09af-42be-b1b4-ef4b31de5792" name="/mnt/x/x.qcow2" pid=1285 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=107

[Luca Lazzeroni (luca-m)]

Il 06/11/2013 21:36, Serge Hallyn ha scritto:
> Interestingly, /etc/apparmor.d/libvirt/libvirt-$uuid.files does have an
> entry allowing rw to the destination path:
>
> "/mnt/x/x.qcow2" rw,
>
> but still I get
>
> Nov 6 21:29:47 kvm-s1 kernel: [ 1432.501141] type=1400 audit(1383769787.515:60): apparmor="STATUS" operation="profile_replace" parent=1382 profile="unconfined" name="libvirt-5d349701-09af-42be-b1b4-ef4b31de5792" pid=1383 comm="apparmor_parser"
> Nov 6 21:29:47 kvm-s1 kernel: [ 1432.502753] type=1400 audit(1383769787.515:61): apparmor="DENIED" operation="open" parent=1 profile="libvirt-5d349701-09af-42be-b1b4-ef4b31de5792" name="/mnt/x/x.qcow2" pid=1285 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=107 ouid=107
>

In fact I've tried to manually add original disk image path with rw
permission and enforce apparmor reload, but system seems to ignore that.
I've compared apparmor profiles with 13.04, but they hadn't change
(except for a few lines related to direct USB device support).

[Serge Hallyn (serge-hallyn)]

Ah! Sorry, I thought this had been fixed, but this appears to be due to
bug 1236455. Using an older raring kernel should fix it for you. What
happens is qemu gets started, then libvirt updates its profile to allow
the access to the new path. The running qemu should get automatically
get the new permissions, but that's not happening.

 duplicate: 1236455

[Luca Lazzeroni (luca-m)]

Ok, but in saucy I cannot use an older raring kernel.

Il giorno 06/nov/2013, alle ore 23:24, Serge Hallyn <email address hidden> ha scritto:

> *** This bug is a duplicate of bug 1236455 ***
> https://bugs.launchpad.net/bugs/1236455
>
> Ah! Sorry, I thought this had been fixed, but this appears to be due to
> bug 1236455. Using an older raring kernel should fix it for you. What
> happens is qemu gets started, then libvirt updates its profile to allow
> the access to the new path. The running qemu should get automatically
> get the new permissions, but that's not happening.
>
> duplicate: 1236455
>
>
> ** This bug has been marked a duplicate of bug 1236455
> Running tasks are not subject to reloaded policies
>
> ** This bug has been marked a duplicate of bug 1236455
> Running tasks are not subject to reloaded policies
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1248577
>
> Title:
> libvirt with securty_driver="apparmor" (default settings) cannot do
> live blockcopy of devices due to permission denied error
>
> Status in “libvirt” package in Ubuntu:
> Confirmed
>
> Bug description:
> Hi,
> if I create a VM using default libvirt settings and try to do a blockcopy of one of its block devices, procedure fails claiming "permission denied" and the original block device looses write permission. Only power-cycling the VM restores correct behaviour.
>
> If I manually edit /etc/libvirt/qemu.conf and set
>
> security_driver="none"
>
> then blockcopy works as expected.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 13.10
> Package: libvirt-bin 1.1.1-0ubuntu8
> ProcVersionSignature: Ubuntu 3.11.0-12.19-generic 3.11.3
> Uname: Linux 3.11.0-12-generic x86_64
> ApportVersion: 2.12.5-0ubuntu2.1
> Architecture: amd64
> Date: Wed Nov 6 15:57:33 2013
> InstallationDate: Installed on 2013-11-04 (2 days ago)
> InstallationMedia: Ubuntu-Server 13.10 "Saucy Salamander" - Release amd64 (20131016)
> MarkForUpload: True
> SourcePackage: libvirt
> UpgradeStatus: No upgrade log present (probably fresh install)
> modified.conffile..etc.apparmor.d.abstractions.libvirt.qemu: [modified]
> modified.conffile..etc.apparmor.d.local.usr.sbin.libvirtd: [modified]
> modified.conffile..etc.libvirt.libvirtd.conf: [modified]
> modified.conffile..etc.libvirt.qemu.conf: [modified]
> modified.conffile..etc.libvirt.qemu.networks.default.xml: [deleted]
> mtime.conffile..etc.apparmor.d.abstractions.libvirt.qemu: 2013-11-06T12:40:14.384226
> mtime.conffile..etc.apparmor.d.local.usr.sbin.libvirtd: 2013-11-06T15:02:46.028029
> mtime.conffile..etc.libvirt.libvirtd.conf: 2013-11-06T11:17:34.844340
> mtime.conffile..etc.libvirt.qemu.conf: 2013-11-06T15:49:54.023964
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1248577/+subscriptions

Ing. Luca Lazzeroni - Trend Servizi Srl
Responsabile R&D
http://www.trendservizi.it