virsh create-snapshot fails to create external snapshot (blockdev-snapshot-sync fails in json monitor)

[Uri Ortov]

When you attempt to create a virtual machine snapshot onto a separate disk, this fails with an error message. The error message is different depending on whether you use snapshot-create or snapshot-create-as but both approaches fail.

The intention is to be able to take a backup of a live VM by freezing the running image file and applying snapshot deltas to a separate file so the backup can save the filesystem image without it being modified during the backup. (i.e. standard practice in VM world)

Format being used is QCOW2 for the disk file which supports snapshots. Manually using qemu-img to create a snapshot on the file does work without any problems.

If you use snapshot-create-as :

snapshot-create-as winxppro3 snapname --disk-only --diskspec hda,snapshot=external,file=/srv/virtual-machines/tester/snap01.qcow2
error: internal error unable to execute QEMU command 'blockdev-snapshot-sync': An undefined error has ocurred

Or if you use snapshot-create and an XML file specification :

virsh # snapshot-create winxppro3 snapspec.xml
error: argument unsupported: unable to handle disk requests in snapshot

snapspec.xml is :

<domainsnapshot>
  <name>snapname</name>
  <description>test</description>
  <disks>
    <disk name='hda' snapshot='external'>
      <source file='/srv/virtual-machines/tester/snap01.qcow2'/>
    </disk>
  </disks>
</domainsnapshot>

PC specification is :

<domain type='kvm' id='2'>
  <name>winxppro3</name>
  <uuid>b6b35d98-dc3a-e03d-031b-906ae079620f</uuid>
  <memory>524288</memory>
  <currentMemory>524288</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.14'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/srv/virtual-machines/tester/winxp2.qcow2'/>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' unit='0'/>
    </disk>

etc. etc.

[Uri Ortov (uri-ortov)]

Just to be clear, this is on 12.04LTS,
 virsh --version
0.9.8
 3.2.0-24-generic #37-Ubuntu SMP Wed Apr 25 08:43:22 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Switching to PC 1.0 spec does not make any difference.

[markpitchless (markpitchless)]

I'm having the same issue on precise 12.04 (upgraded from lucid):

 # virsh snapshot-create-as lucid-server backup --disk-only
 error: internal error unable to execute QEMU command 'blockdev-snapshot-sync': An undefined error has ocurred

libvirt log shows the same:

 2012-06-07 14:54:46.723+0000: 3435: error : qemuMonitorJSONCheckError:318 : internal error unable to execute QEMU command 'blockdev-snapshot-sync': An undefined error has ocurred

Running Precise: 3.2.0-24-generic #39-Ubuntu SMP Mon May 21 16:52:17 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
# virsh --version
0.9.8

Same as Uri the virt machine has a qcow2 disk that I can snapshot ok manually using qemu-img.

    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/machines/lucid-server.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>

[markpitchless (markpitchless)]

Note that the command I posted should (I believe) have created an internal snapshot, so this bug seems to effect both internal and external snapshots.

[Mark - Syminet (mark-syminet)]

Same exact errors here.

[Julian Held (julian-held)]

I think it's apparmor, I had an similar error and it went away by deactivating the corresponding apparmor profile.
Check your syslog for apparmor messages please.

[Davide Guerri (davide-guerri)]

Thanks Murrayy, actually it's apparmour the problem:

the --disk-only option of snapshot-create[-as] only supports external snapshot so it tries to create a new file which name matches no apparmour regexps in the corresponding profile.

[Serge Hallyn (serge-hallyn)]

Thanks, Davide, cann you tell us the paths of the snapshot files which need to be whitelisted?

[Davide Guerri (davide-guerri)]

The path, excluding extension, is the same as the original image.
Unfortunately the extension can be chosen by the user:

"Name
The name for this snapshot. If the name is specified when initially creating the snapshot, then the snapshot will have that particular name. If the name is omitted when initially creating the snapshot, then libvirt will make up a name for the snapshot, based on the time when it was created."
  (from http://libvirt.org/formatsnapshot.html) with the name parameter)

Assuming my_dom disk image is stored under /nfs/diskimages/my_dom.img, a command like the following:

# snapshot-create-as my_dom my_snap --disk-only

will create /nfs/diskimages/my_dom.my_snap changing domain definition XML to use this file instead of /nfs/diskimages/my_dom.img (/nfs/diskimages/my_dom.img will be a backing file for /nfs/diskimages/my_dom.my_snap)

However I fear it's not that simple because even if I try to use a snapshot name like mysnap.img, snaphot still fails because the original image name is removed from the apparmour profile dinamically created/maintained by libvirt under /etc/apparmor.d/libvirt. The original filename is replaced with the new image name.

So, to sum up, I think the following might be needed in order to make disk-only snapshot work

 1) virt-aa-helper (/etc/apparmor.d/usr.lib.libvirt.virt-aa-helper) should be able to read virtual machine image files even if the extension isn't imq/qcow2/...
 2) dynamically created profiles for libvirt (/etc/apparmor.d/libvirt/libvirt-XXXX.files) should retain the old image filename

Please, be aware that after the "snapshot-create" command fails, the corresponding profile under /etc/apparmor.d/libvirt/ isn't coherent anymore with the real filename for virtual images.

[Davide Guerri (davide-guerri)]

also present in Ubuntu 12.10 Quantal

[Ilya Ryabinin (riv1329)]

also present in Ubuntu 13.04 Raring

[Serge Hallyn (serge-hallyn)]

 affects ubuntu/libvirt
 status confirmed
 priority medium

[Theodor van Nahl (tvn87)]

Is there any bugfix in sight or work around known except for disabling apparmor?

[Riccardo Casatta (riccardo-casatta)]

i don't know others work around

2013/5/17 Theodor van Nahl <email address hidden>

> Is there any bugfix in sight or work around known except for disabling
> apparmor?
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1096125).
> https://bugs.launchpad.net/bugs/1004606
>
> Title:
> virsh create-snapshot fails to create external snapshot (blockdev-
> snapshot-sync fails in json monitor)
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1004606/+subscriptions
>

--
Riccardo Casatta

[Theodor van Nahl (tvn87)]

I found a that looks better than deactivating apparmor.

I found here ( http://libvirt.org/drvqemu.html#securitysvirtaa ) the information that Apparmor is „just“ used for protecting the vm host and that there is a TEMPLATE under /etc/apparmor.d/libvirt/ that can be modified. In that TEMPLATE I included one of my own rules and under /etc/apparmor.d/local/usr.sbin.libvirtd i added a similar rule.

[Shweta Salaria (salaria-shweta)]

Theodor, can you please tell the rules that you added under TEMPLATE and /etc/apparmor.d/local/usr.sbin.libvirt?

Thanks !

[Tobias Bradtke (webwurst)]

Yes Theodor, please share your apparmor-rules :)

For the moment i am following this guide to disable just the libvirt-apparmor-profile:

  http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.0.2/html/Installation_Guide/hypervisor-kvm-install-flow.html#hypervisor-host-install-security-policies "Configure Apparmor (Ubuntu)"

  $ sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/
  $ sudo ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/
  $ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd
  $ sudo apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper

Stop running virtual machines.

  $ sudo service apparmor restart
  $ sudo apparmor_status

Start machines again.

[Theodor van Nahl (tvn87)]

In the /etc/apparmor.d/local/usr.sbin.libvirtd file I just created one rule to give libvirtd read'n'write access to the images in my storage pool with the following line:

            "/var/lib/libvirt/images/*.img" rw,

As preliminary: I have created my own naming convention for my overlays, these are used for incremental backups to another server. This convention can be found in my abstractation and has to be adjusted to your own needs.

First of all I've created my own abstraction as /etc/apparmor.d/local/abstraction-libvirt-storage. This file gives the clients access to the important images like that:

        "/var/lib/libvirt/images/*.base.img" rw,
        "/var/lib/libvirt/images/*.base.img" rw,
        "/var/lib/libvirt/images/*.stable_overlay.img" rw,
        "/var/lib/libvirt/images/*.running.img" rw,

The /etc/apparmor.d/libvirt/TEMPLATE file is a source for all rule files in /etc/apparmor.d/libvirt/. There you need to source the abstraction-libvirt-storage so the TEMPLATE looks similar to this one (adjust to your own needs):

profile LIBVIRT_TEMPLATE {
  #include <abstractions/libvirt-qemu>
  #include <local/abstractation-libvirt-storage>
}

It is also possible to put the information of the abstraction-libvirt-storage file directly into the TEMPLATE but a change on some of the rules would require to edit multiple files ( /etc/apparmor.d/libvirt/*)

I hope this will help. This adjustments should be fine for safety requirement, because the host should still be secured against guests and thats the only thing you can do with libvirt+apparmor.

[Serge Hallyn (serge-hallyn)]

I believe this is fixed at least in trusty (have not tried elsewhere). At least I can create external snapshots of a stopped domain, then start the domain. (The snapshot image is listed in /etc/apparmor.d/libvirt/libvirt-$uuid.files).

I will mark this fix committed, if it is still broken for (in trusty) you then please reply.

I will mark it as affecting precise. If there are other releases where it is still valid please let me know.

[Stefan Reiff (kroko87)]

Also trusty is affected!

[Stefan Reuter (stefan-reuter)]

Trusty is still affected if I try to make a snapshot from a running domain.

[Serge Hallyn (serge-hallyn)]

@Stefan,

can you please show the exacty command you are using and how it fails?

[Jonathan Millett (t-jon)]

Failing for me as wel on trusty when snappshotting a running domain apparently due to app armour.

virsh # list
 Id Name State
----------------------------------------------------
 7 jam1 running

virsh # domblklist jam1
Target Source
------------------------------------------------
hda /var/lib/libvirt/images/jam1/tmprK58RK.qcow2

$ sudo qemu-img info /var/lib/libvirt/images/jam1/tmprK58RK.qcow2
image: /var/lib/libvirt/images/jam1/tmprK58RK.qcow2
file format: qcow2
virtual size: 5.0G (5368709120 bytes)
disk size: 613M
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false

virsh # snapshot-create-as jam1 jam1-A --disk-only --atomic
error: internal error: unable to execute QEMU command 'transaction': Could not open '/var/lib/libvirt/images/jam1/tmprK58RK.qcow2': Could not open '/var/lib/libvirt/images/jam1/tmprK58RK.qcow2': Permission denied: Permission denied

$ ls -al /var/lib/libvirt/images/jam1/tmprK58RK.qcow2
-rw-r--r-- 1 libvirt-qemu kvm 642908160 Jun 24 13:18 /var/lib/libvirt/images/jam1/tmprK58RK.qcow2

/var/log/syslog
Jun 24 13:22:51 bat kernel: [14013.429506] type=1400 audit(1403630571.119:76): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-dafd2c09-a81d-4ee3-a95c-beb50aecf4e8" pid=17961 c\
omm="apparmor_parser"
Jun 24 13:22:51 bat kernel: [14013.435384] type=1400 audit(1403630571.127:77): apparmor="DENIED" operation="open" profile="libvirt-dafd2c09-a81d-4ee3-a95c-beb50aecf4e8" name="/var/lib/libvirt/images/jam1/tmprK\
58RK.qcow2" pid=15483 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=124 ouid=124
Jun 24 13:22:51 bat kernel: [14013.435401] type=1400 audit(1403630571.127:78): apparmor="DENIED" operation="open" profile="libvirt-dafd2c09-a81d-4ee3-a95c-beb50aecf4e8" name="/var/lib/libvirt/images/jam1/tmprK\
58RK.qcow2" pid=15483 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=124 ouid=124
Jun 24 13:22:51 bat kernel: [14013.435437] type=1400 audit(1403630571.127:79): apparmor="DENIED" operation="open" profile="libvirt-dafd2c09-a81d-4ee3-a95c-beb50aecf4e8" name="/var/lib/libvirt/images/jam1/tmprK\
58RK.qcow2" pid=15483 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=124 ouid=124
Jun 24 13:22:51 bat kernel: [14013.566334] type=1400 audit(1403630571.259:80): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-dafd2c09-a81d-4ee3-a95c-beb50aecf4e8" pid=17964 c\
omm="apparmor_parser"

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty

$ uname -a
Linux bat 3.13.0-29-generic #53-Ubuntu SMP Wed Jun 4 21:00:20 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

$ virsh --version
1.2.2

[Serge Hallyn (serge-hallyn)]

Thanks.

So in your case, because there is a separate directory fo rimages for your domain, it would be safe to add

/var/lib/libvirt/images/jam1/*.qcow2 rw,

to your /etc/apparmor.d/libvirt/libvirt-dafd2c09-a81d-4ee3-a95c-beb50aecf4e8.files

However, in cases where the domain's disk image is a file under /var/lib/libvirt/images/ itself, this is not safe.

Can you confirm that adding the above line to that file works around the issue for you?

[NightShade (tim-night-shade)]

Definitely still a problem in utopic, I think the issue is caused by this part of the apparmor config:

/var/lib/libvirt/images/openwrt-x86-kvm_guest-combined-ext4.img-zfs-1.qcow2 is my current snapshot and I am trying to commit it back to /var/lib/libvirt/images/openwrt-x86-kvm_guest-combined-ext4.img which is set as read only by apparmor

root@desktop-tim:/var/lib/libvirt/images# cat /etc/apparmor.d/libvirt/libvirt-8c73d931-5b74-266c-bb09-cb3e16cb9eb0.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
  "/var/log/libvirt/**/OpenWRT.log" w,
  "/var/lib/libvirt/**/OpenWRT.monitor" rw,
  "/var/run/libvirt/**/OpenWRT.pid" rwk,
  "/run/libvirt/**/OpenWRT.pid" rwk,
  "/var/run/libvirt/**/*.tunnelmigrate.dest.OpenWRT" rw,
  "/run/libvirt/**/*.tunnelmigrate.dest.OpenWRT" rw,
  "/var/lib/libvirt/images/openwrt-x86-kvm_guest-combined-ext4.img-zfs-1.qcow2" rw,
  "/var/lib/libvirt/images/openwrt-x86-kvm_guest-combined-ext4.img" r,
  # don't audit writes to readonly files
  deny "/var/lib/libvirt/images/openwrt-x86-kvm_guest-combined-ext4.img" w,
  /dev/vhost-net rw,
  "/var/lib/libvirt/images/openwrt-x86-kvm_guest-combined-ext4.img" rw,

[NightShade (tim-night-shade)]

Experimenting further if I take out the seclabel line (shown below) from my VM definition everything works as expected with snapshots.

  <seclabel type='dynamic' model='apparmor' relabel='yes'/>

[NightShade (tim-night-shade)]

I think I have found the cause of this problem, in bug number 453335 an apparmor fix was put in place to deny access to certain read only media and also to hide the error message about access being denied.

I think this is causing the duplicate entries in apparmor and also hiding the errors about access to the snapshot file.

[NightShade (tim-night-shade)]

I believe this is the problematic git commit: http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=d0d4b8ad76d3e8a859ee90701a21a3f003a22c1f;hp=dae7054b7697d30f2b88cd5dff3dfb954323d40b

[Jamie Strandboge (jdstrand)]

There does seem to be a bug in there. Couple of points:
1. we do want to enforce that the media is readonly if libvirt says it is (hence the explicit deny)
2. we don't want to grant 'w' access in one line, only to take it away in an explicit deny
3. I don't know what 'relabel' is supposed to mean in the context of apparmor

So virt-aa-helper needs to refine its logic. The referenced commit isn't the actual problem though-- that bug was about when <readonly/> was present, qemu would try to open rw but apparmor would log the harmless denial. The commit simply silenced logging for a denial that was happening anyway. This bug is about applying that deny rule at the wrong time.

[Steve Langasek (vorlon)]

The Precise Pangolin has reached end of life, so this bug will not be fixed for that release