Ubuntu

Apparmor prevents KVM tunnelled migration

Reported by Simon Déziel on 2011-10-06
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
High
Unassigned
Lucid
Undecided
Unassigned
Maverick
Undecided
Unassigned
Natty
Undecided
Unassigned
Oneiric
Undecided
Unassigned

Bug Description

=======================================
SRU Justification:
1. Impact: tunnelled migration fails
2. Development fix: adjust the apparmor security driver in libvirt to allow guests the access to the tunneled migration info
3. Stable fix: same as development fix
4. Test case:
 1. install libvirt-bin on two machines (sourcehost and targethost) sharing (nfs - use -o nfsvers=3 on oneiric) read-write storage
 2. set up a kvm guest on the shared storage, start it on sourcehost
 3. create rsa key on sourcehost, and put it into root's authorized_keys on targethost.
 4. stop and start libvirt-bin on both hosts
 5. as root, ssh from sourcehost to targethost, then log out (to answer the known-host question which otherwise makes libvirt fail to connect)
 6. virsh migrate --live --p2p --tunnelled guestvm qemu+ssh://targethost/system
 7. regression potential: if the policy change was bad, it could cause libvirt guests to receive too much privilege.
=======================================

While attempting a live migration the destination host (node1) logs this :

Oct 5 17:13:56 node1 kernel: [ 1418.872987] type=1503 audit(1317849236.311:29): operation="mknod" pid=1975 parent=1 profile="libvirt-4aa60863-6b03-2f19-897f-4de6d12c96e1" requested_mask="c::" denied_mask="c::" fsuid=0 ouid=0 name="/var/
run/libvirt/qemu/qemu.tunnelmigrate.dest.guest1"

The source system was running this command :

root@node2:~# virsh migrate --live --p2p --tunnelled guest1 qemu+ssh://192.168.88.51/system

Both systems are running the same distro and package versions :

# lsb_release -rd
Description: Ubuntu 10.04.3 LTS
Release: 10.04

# apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 0.7.5-5ubuntu27.16
  Candidate: 0.7.5-5ubuntu27.16
  Version table:
 *** 0.7.5-5ubuntu27.16 0
        500 http://archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
        500 http://archive.ubuntu.com/ubuntu/ lucid-security/main Packages
        100 /var/lib/dpkg/status
     0.7.5-5ubuntu27 0
        500 http://archive.ubuntu.com/ubuntu/ lucid/main Packages

Related branches

Simon Déziel (sdeziel) wrote :

Adding the following to /etc/apparmor.d/abstractions/libvirt-qemu :

  /{,var/}run/libvirt/qemu/* rw,

After fixing the Apparmor profile, the migration still fails but not due to Apparmor getting in the way. Instead it blocks on another bug (that deserves a separated bug in LP) :

Oct 6 17:30:13 node2 libvirtd: 17:30:13.139: error : qemuMonitorTextGetMigrationStatus:982 : internal error cannot parse migration data transferred statistic 1052 kbytes#015#012remaining ram: 539972 kbytes#015#012total ram: 541024 kbytes#015#012

Note that using a regular migration (no "--p2p --tunnelled" flags) works.

Simon Déziel (sdeziel) on 2011-10-06
summary: - KVM migration fails when tunnelled
+ Apparmor prevents KVM tunnelled migration
Simon Déziel (sdeziel) wrote :

The bug #869590 (https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/869590) was opened for the qemuMonitrText migration parsing error.

Serge Hallyn (serge-hallyn) wrote :

Thanks for submitting this bug and the detailed info.

Jamie, the needed rule sounds dangerous to me. Can you comment?

Changed in libvirt (Ubuntu):
importance: Undecided → High
assignee: nobody → Serge Hallyn (serge-hallyn)
assignee: Serge Hallyn (serge-hallyn) → Jamie Strandboge (jdstrand)
Jamie Strandboge (jdstrand) wrote :

This is not an appropriate rule because it negates the property of guest isolation. This should be handled dynamically by the security driver, but it is not (which is the bug). This could be done in a couple of ways. What are the contents of /var/log/libvirt/qemu/guest1.log after the AppArmor denial?

Changed in libvirt (Ubuntu):
status: New → Incomplete
Simon Déziel (sdeziel) wrote :

On the receiving host :

root@node1:~# cat /var/log/libvirt/qemu/guest1.log
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 512 -smp 1 -name guest1 -uuid 4aa60863-6b03-2f19-897f-4de6d12c96e1 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/guest1.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/var/lib/libvirt/images/guest1.img,if=virtio,index=0,boot=on,format=raw,cache=writeback -net nic,macaddr=52:54:00:03:d7:15,vlan=0,model=virtio,name=virtio.0 -net tap,fd=36,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 127.0.0.1:0 -vga cirrus -incoming unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.guest1
char device redirected to /dev/pts/1
bind(unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.guest1): Permission denied

Simon Déziel (sdeziel) wrote :

@Jamie, please let me know if more information is required. I've set the status to "New" again but let me know if I should have set it to "Triaged" or any other status. Thank you.

Changed in libvirt (Ubuntu):
status: Incomplete → New
Jamie Strandboge (jdstrand) wrote :

A workaround for this particular VM is to add the following to /etc/apparmor.d/libvirt/libvirt-4aa60863-6b03-2f19-897f-4de6d12c96e1 (note that I expressly did *not* use /etc/apparmor.d/libvirt/libvirt-4aa60863-6b03-2f19-897f-4de6d12c96e1.files):
 /var/run/libvirt/qemu/qemu.tunnelmigrate.dest.guest1 rw,

Changed in libvirt (Ubuntu):
status: New → Triaged
Simon Déziel (sdeziel) wrote :

@Jamie, I assumed that your advise to not modify the .files profile directly was because this file is regenerated on the VM startup. I have attached a patch to have virt-aa-helper add the required rule to the /etc/apparmor.d/libvirt/libvirt-<UUID>.files. Let me know if that's not the correct/best way to handle this. Thank you.

The attachment "adjust virt-aa-helper to allow tunnelled migrations" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Jamie Strandboge (jdstrand) wrote :

Simon,

That's correct. Without testing it, this patch looks fine. Ideally this would be handled better via the security driver framework, but the latest git seems to indicate this isn't implemented yet, so this would be fine for Precise.

So getting this into Ubuntu, the patch should be tested, submitted upstream and applied to the Ubuntu development release (Precise). Once that is done, it can be backported to earlier releases as part of the SRU process.

Thanks for your patch!

Serge Hallyn (serge-hallyn) wrote :

@Simon,

I will send apply/test this patch on precise and send the result upstream in the next few days. If you were intending to do that yourself, please let me know and sorry for stepping on your toes :)

Thanks again for your patch.

> If you were intending to do that yourself, please let me know and sorry for stepping on your toes :)

Oh no, be my guest. Thanks a lot for taking care of this (as well as LP:
#869590).

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 0.9.7-2ubuntu2

---------------
libvirt (0.9.7-2ubuntu2) precise; urgency=low

  * apparmor-allow-tunnelled-migration.patch: adjust virt-aa-helper to
    allow tunnelled migration (LP: #869553)
    (Thanks to Simon Deziel <email address hidden>)
  * fix break in last upload: *really* don't apply patch
    debian/Don-t-enable-default-network-on-boot.patch
 -- Serge Hallyn <email address hidden> Fri, 18 Nov 2011 15:52:38 -0600

Changed in libvirt (Ubuntu):
status: Triaged → Fix Released
Serge Hallyn (serge-hallyn) wrote :

While this patch sufficed in lucid, it did not in precise. We'll have to figure out what else is needed so that we can send a working patch upstream.

Changed in libvirt (Ubuntu):
status: Fix Released → Confirmed
Serge Hallyn (serge-hallyn) wrote :

2011-11-18 23:43:36.939: 4980: debug : qemuProcessStart:3129 : setting security label on pipe used for migration
 2011-11-18 23:43:36.940: 4980: error : AppArmorSetImageFDLabel:795 : internal error could not find path for descriptor

is what i get in the logs on precise.

Serge Hallyn (serge-hallyn) wrote :

@Jamie,

With this debdiff I am able to do tunnelled migration on precise.

I think it's ok to do, but I'll wait for you to take a look before sending it upstream and pushing to the archive.

Jamie Strandboge (jdstrand) wrote :

Serge,

Sorry for the delay.

This patch seems fine assuming that the behavior is expected (ie, the path is deleted intentionally). I think I'd prefer to use VIR_WARN() instead of virSecurityReportError(). Perhaps:
   VIR_WARN("could not find path for descriptor %s, skipping", proc);

IIRC, the use of virSecurityReportError() stops error reporting and may mask other problems, so we only want to warn here. If the removed path behavior is expected, getting it upstream (using VIR_WARN()) should not be a problem.

Changed in libvirt (Ubuntu):
assignee: Jamie Strandboge (jdstrand) → nobody
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 0.9.7-2ubuntu4

---------------
libvirt (0.9.7-2ubuntu4) precise; urgency=low

  * debian/apparmor/usr.sbin.libvirtd:
    - allow access to /etc/libvirt/hooks/** (LP: #891472)
    - #include <local/usr.sbin.libvirtd> for site-local customizations
  * debian/control: Suggest cgroup-lite | cgroup-bin (LP: #544146)
  * debian/patches/ubuntu/apparmor-allow-tunnelled-migration-2.patch:
    Warn but don't error out when we can't find a pathname for a file.
    This is needed to support tunnelled migration. (LP: #869553)
 -- Serge Hallyn <email address hidden> Fri, 02 Dec 2011 11:50:47 -0600

Changed in libvirt (Ubuntu):
status: Confirmed → Fix Released
description: updated
description: updated
Changed in libvirt (Ubuntu Maverick):
status: New → Won't Fix
Changed in libvirt (Ubuntu Natty):
status: New → Won't Fix
description: updated
Level15 (ejortegau) wrote :

I am running Precise but the bug continues to affect me, despite it being marked as solved. This is what I get on libvirt.log:

2013-11-01 12:10:57.180+0000: 21958: warning : AppArmorSetImageFDLabel:799 : could not find path for descriptor /proc/self/fd/21, skipping
2013-11-01 12:10:57.430+0000: 21957: error : qemuMonitorIO:603 : internal error End of file from monitor
2013-11-01 12:10:57.955+0000: 21957: error : virFDStreamWrite:330 : cannot write to stream: Broken pipe
2013-11-01 12:10:57.956+0000: 21957: warning : daemonStreamEvent:139 : event for client=0x16360d0 stream st=0x7f97a40008e0, but missing stream state

libvirt-bin version is 0.9.8-2ubuntu17.13. I have tried fiddling around with apparmor profiles to workaround it with no luck. It might be worth mentioning that the destination directory of the VM image is not the default.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in libvirt (Ubuntu Lucid):
status: New → Confirmed
Changed in libvirt (Ubuntu Oneiric):
status: New → Confirmed
Serge Hallyn (serge-hallyn) wrote :

@Level15,

please file a new bug, as yours is different from what this bug fixed.

Changed in libvirt (Ubuntu Oneiric):
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers