Apparmor prevents KVM tunnelled migration

Bug #869553 reported by Simon Déziel
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Fix Released
High
Unassigned
Lucid
Won't Fix
Undecided
Unassigned
Maverick
Won't Fix
Undecided
Unassigned
Natty
Won't Fix
Undecided
Unassigned
Oneiric
Invalid
Undecided
Unassigned

Bug Description

=======================================
SRU Justification:
1. Impact: tunnelled migration fails
2. Development fix: adjust the apparmor security driver in libvirt to allow guests the access to the tunneled migration info
3. Stable fix: same as development fix
4. Test case:
 1. install libvirt-bin on two machines (sourcehost and targethost) sharing (nfs - use -o nfsvers=3 on oneiric) read-write storage
 2. set up a kvm guest on the shared storage, start it on sourcehost
 3. create rsa key on sourcehost, and put it into root's authorized_keys on targethost.
 4. stop and start libvirt-bin on both hosts
 5. as root, ssh from sourcehost to targethost, then log out (to answer the known-host question which otherwise makes libvirt fail to connect)
 6. virsh migrate --live --p2p --tunnelled guestvm qemu+ssh://targethost/system
 7. regression potential: if the policy change was bad, it could cause libvirt guests to receive too much privilege.
=======================================

While attempting a live migration the destination host (node1) logs this :

Oct 5 17:13:56 node1 kernel: [ 1418.872987] type=1503 audit(1317849236.311:29): operation="mknod" pid=1975 parent=1 profile="libvirt-4aa60863-6b03-2f19-897f-4de6d12c96e1" requested_mask="c::" denied_mask="c::" fsuid=0 ouid=0 name="/var/
run/libvirt/qemu/qemu.tunnelmigrate.dest.guest1"

The source system was running this command :

root@node2:~# virsh migrate --live --p2p --tunnelled guest1 qemu+ssh://192.168.88.51/system

Both systems are running the same distro and package versions :

# lsb_release -rd
Description: Ubuntu 10.04.3 LTS
Release: 10.04

# apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 0.7.5-5ubuntu27.16
  Candidate: 0.7.5-5ubuntu27.16
  Version table:
 *** 0.7.5-5ubuntu27.16 0
        500 http://archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
        500 http://archive.ubuntu.com/ubuntu/ lucid-security/main Packages
        100 /var/lib/dpkg/status
     0.7.5-5ubuntu27 0
        500 http://archive.ubuntu.com/ubuntu/ lucid/main Packages

Tags: patch

Related branches

Revision history for this message
Simon Déziel (sdeziel) wrote :

Adding the following to /etc/apparmor.d/abstractions/libvirt-qemu :

  /{,var/}run/libvirt/qemu/* rw,

After fixing the Apparmor profile, the migration still fails but not due to Apparmor getting in the way. Instead it blocks on another bug (that deserves a separated bug in LP) :

Oct 6 17:30:13 node2 libvirtd: 17:30:13.139: error : qemuMonitorTextGetMigrationStatus:982 : internal error cannot parse migration data transferred statistic 1052 kbytes#015#012remaining ram: 539972 kbytes#015#012total ram: 541024 kbytes#015#012

Note that using a regular migration (no "--p2p --tunnelled" flags) works.

Simon Déziel (sdeziel)
summary: - KVM migration fails when tunnelled
+ Apparmor prevents KVM tunnelled migration
Revision history for this message
Simon Déziel (sdeziel) wrote :

The bug #869590 (https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/869590) was opened for the qemuMonitrText migration parsing error.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks for submitting this bug and the detailed info.

Jamie, the needed rule sounds dangerous to me. Can you comment?

Changed in libvirt (Ubuntu):
importance: Undecided → High
assignee: nobody → Serge Hallyn (serge-hallyn)
assignee: Serge Hallyn (serge-hallyn) → Jamie Strandboge (jdstrand)
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

This is not an appropriate rule because it negates the property of guest isolation. This should be handled dynamically by the security driver, but it is not (which is the bug). This could be done in a couple of ways. What are the contents of /var/log/libvirt/qemu/guest1.log after the AppArmor denial?

Changed in libvirt (Ubuntu):
status: New → Incomplete
Revision history for this message
Simon Déziel (sdeziel) wrote :

On the receiving host :

root@node1:~# cat /var/log/libvirt/qemu/guest1.log
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 512 -smp 1 -name guest1 -uuid 4aa60863-6b03-2f19-897f-4de6d12c96e1 -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/guest1.monitor,server,nowait -monitor chardev:monitor -boot c -drive file=/var/lib/libvirt/images/guest1.img,if=virtio,index=0,boot=on,format=raw,cache=writeback -net nic,macaddr=52:54:00:03:d7:15,vlan=0,model=virtio,name=virtio.0 -net tap,fd=36,vlan=0,name=tap.0 -chardev pty,id=serial0 -serial chardev:serial0 -parallel none -usb -vnc 127.0.0.1:0 -vga cirrus -incoming unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.guest1
char device redirected to /dev/pts/1
bind(unix:/var/run/libvirt/qemu/qemu.tunnelmigrate.dest.guest1): Permission denied

Revision history for this message
Simon Déziel (sdeziel) wrote :

@Jamie, please let me know if more information is required. I've set the status to "New" again but let me know if I should have set it to "Triaged" or any other status. Thank you.

Changed in libvirt (Ubuntu):
status: Incomplete → New
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

A workaround for this particular VM is to add the following to /etc/apparmor.d/libvirt/libvirt-4aa60863-6b03-2f19-897f-4de6d12c96e1 (note that I expressly did *not* use /etc/apparmor.d/libvirt/libvirt-4aa60863-6b03-2f19-897f-4de6d12c96e1.files):
 /var/run/libvirt/qemu/qemu.tunnelmigrate.dest.guest1 rw,

Changed in libvirt (Ubuntu):
status: New → Triaged
Revision history for this message
Simon Déziel (sdeziel) wrote :

@Jamie, I assumed that your advise to not modify the .files profile directly was because this file is regenerated on the VM startup. I have attached a patch to have virt-aa-helper add the required rule to the /etc/apparmor.d/libvirt/libvirt-<UUID>.files. Let me know if that's not the correct/best way to handle this. Thank you.

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "adjust virt-aa-helper to allow tunnelled migrations" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

tags: added: patch
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Simon,

That's correct. Without testing it, this patch looks fine. Ideally this would be handled better via the security driver framework, but the latest git seems to indicate this isn't implemented yet, so this would be fine for Precise.

So getting this into Ubuntu, the patch should be tested, submitted upstream and applied to the Ubuntu development release (Precise). Once that is done, it can be backported to earlier releases as part of the SRU process.

Thanks for your patch!

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@Simon,

I will send apply/test this patch on precise and send the result upstream in the next few days. If you were intending to do that yourself, please let me know and sorry for stepping on your toes :)

Thanks again for your patch.

Revision history for this message
Simon Déziel (sdeziel) wrote : Re: [Bug 869553] Re: Apparmor prevents KVM tunnelled migration

> If you were intending to do that yourself, please let me know and sorry for stepping on your toes :)

Oh no, be my guest. Thanks a lot for taking care of this (as well as LP:
#869590).

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 0.9.7-2ubuntu2

---------------
libvirt (0.9.7-2ubuntu2) precise; urgency=low

  * apparmor-allow-tunnelled-migration.patch: adjust virt-aa-helper to
    allow tunnelled migration (LP: #869553)
    (Thanks to Simon Deziel <email address hidden>)
  * fix break in last upload: *really* don't apply patch
    debian/Don-t-enable-default-network-on-boot.patch
 -- Serge Hallyn <email address hidden> Fri, 18 Nov 2011 15:52:38 -0600

Changed in libvirt (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

While this patch sufficed in lucid, it did not in precise. We'll have to figure out what else is needed so that we can send a working patch upstream.

Changed in libvirt (Ubuntu):
status: Fix Released → Confirmed
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

2011-11-18 23:43:36.939: 4980: debug : qemuProcessStart:3129 : setting security label on pipe used for migration
 2011-11-18 23:43:36.940: 4980: error : AppArmorSetImageFDLabel:795 : internal error could not find path for descriptor

is what i get in the logs on precise.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@Jamie,

With this debdiff I am able to do tunnelled migration on precise.

I think it's ok to do, but I'll wait for you to take a look before sending it upstream and pushing to the archive.

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Serge,

Sorry for the delay.

This patch seems fine assuming that the behavior is expected (ie, the path is deleted intentionally). I think I'd prefer to use VIR_WARN() instead of virSecurityReportError(). Perhaps:
   VIR_WARN("could not find path for descriptor %s, skipping", proc);

IIRC, the use of virSecurityReportError() stops error reporting and may mask other problems, so we only want to warn here. If the removed path behavior is expected, getting it upstream (using VIR_WARN()) should not be a problem.

Changed in libvirt (Ubuntu):
assignee: Jamie Strandboge (jdstrand) → nobody
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package libvirt - 0.9.7-2ubuntu4

---------------
libvirt (0.9.7-2ubuntu4) precise; urgency=low

  * debian/apparmor/usr.sbin.libvirtd:
    - allow access to /etc/libvirt/hooks/** (LP: #891472)
    - #include <local/usr.sbin.libvirtd> for site-local customizations
  * debian/control: Suggest cgroup-lite | cgroup-bin (LP: #544146)
  * debian/patches/ubuntu/apparmor-allow-tunnelled-migration-2.patch:
    Warn but don't error out when we can't find a pathname for a file.
    This is needed to support tunnelled migration. (LP: #869553)
 -- Serge Hallyn <email address hidden> Fri, 02 Dec 2011 11:50:47 -0600

Changed in libvirt (Ubuntu):
status: Confirmed → Fix Released
description: updated
description: updated
Changed in libvirt (Ubuntu Maverick):
status: New → Won't Fix
Changed in libvirt (Ubuntu Natty):
status: New → Won't Fix
description: updated
Revision history for this message
Level15 (ejortegau) wrote :

I am running Precise but the bug continues to affect me, despite it being marked as solved. This is what I get on libvirt.log:

2013-11-01 12:10:57.180+0000: 21958: warning : AppArmorSetImageFDLabel:799 : could not find path for descriptor /proc/self/fd/21, skipping
2013-11-01 12:10:57.430+0000: 21957: error : qemuMonitorIO:603 : internal error End of file from monitor
2013-11-01 12:10:57.955+0000: 21957: error : virFDStreamWrite:330 : cannot write to stream: Broken pipe
2013-11-01 12:10:57.956+0000: 21957: warning : daemonStreamEvent:139 : event for client=0x16360d0 stream st=0x7f97a40008e0, but missing stream state

libvirt-bin version is 0.9.8-2ubuntu17.13. I have tried fiddling around with apparmor profiles to workaround it with no luck. It might be worth mentioning that the destination directory of the VM image is not the default.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in libvirt (Ubuntu Lucid):
status: New → Confirmed
Changed in libvirt (Ubuntu Oneiric):
status: New → Confirmed
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

@Level15,

please file a new bug, as yours is different from what this bug fixed.

Changed in libvirt (Ubuntu Oneiric):
status: Confirmed → Invalid
Revision history for this message
Rolf Leggewie (r0lf) wrote :

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".

Changed in libvirt (Ubuntu Lucid):
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.