libvirt does not grant qemu-guest-agent channel perms

[Mark Grocock]

[Impact]

 * If one defines guest channels manually (xml) or via tools like virt-
   manager (there it defaults to add channels for some distros), then
   starting the guest fails.
   There are two reason:
   1. by default the base dir for the channels doesn't exists so the
      open fails
   2. further virt-aa-helper does not create a matchign rule to allow
      access, so apparmor blocks

 * In latter versions the paths are slightly different (better namespaced
   by guest name), but still similar. So this still can be considered
   backporting the virt-aa-helper change, and making sure the base dir
   exists (only needed in this old release) is a postinst change.

[Test Case]

 * Create a libvirt based KVM guest on Trusty the way you prefer
 * Add a guest channel to it by adding a snippet like:
    <channel type='unix'>
      <source mode='bind' />
      <target type='virtio' name='org.qemu.guest_agent.0'/>
    </channel>
 * Start the guest via e.g. virsh
 * Without the fix this fails, you'll see in strace a failed call to open
   the channel, but even if e.g. dirs are created then apparmor will block
   the access.
 * With the fix installed the guest starts correctly

[Regression Potential]

 * The patch is a backport and only a slight change to code that is used
   quite some time (paths were different in Trusty). In any case it is
   "adding" one more rule to open up apparmor. It should functionally not
   regress by that, if anything one could consider it security risk, but
   due to the guestname-namespacing in the rule now generated this shoudl
   be safe - see the tail of comment #58 for some considerations on that.

 * The postinst change only runs if the dir is not existing, which should
   ensure that no former unexpected setup makes the postinst fail

[Other Info]

 * Tests on the issue itself look good based on a ppa, see comment #59

----

=======================================
1. Impact: cannot create a default RHEL7 vm in virt-manager
2. fix: allow use of qemu-guest-agent channel
3. test case: see in description below. Create a VM in virt-manager specifying
   Linux os and RHEL7.
4. Regression potential: there should be none. We are only adding an
   apparmor permission for unix sockets which libvirt creates when needed
   for kvm vms.
=======================================

Create a new VM, choose Linux for OS type and Red Hat Enterprise Linux 7 (or later) for Version. Proceed through the wizard leaving all other options unchanged. On clicking Finish, the following error is displayed:

Unable to complete install: 'internal error: process exited while connecting to monitor: 2014-11-18T16:00:11.802430Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/rhel7.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: No such file or directory
2014-11-18T16:00:11.802483Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/rhel7.org.qemu.guest_agent.0,server,nowait: chardev: opening backend "socket" failed
'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 91, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1820, in do_install
    guest.start_install(meter=meter)
  File "/usr/share/virt-manager/virtinst/guest.py", line 403, in start_install
    noboot)
  File "/usr/share/virt-manager/virtinst/guest.py", line 467, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3398, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: process exited while connecting to monitor: 2014-11-18T16:00:11.802430Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/rhel7.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: No such file or directory
2014-11-18T16:00:11.802483Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/rhel7.org.qemu.guest_agent.0,server,nowait: chardev: opening backend "socket" failed

ProblemType: Bug
DistroRelease: Ubuntu 14.10
Package: virt-manager 1:1.0.1-0ubuntu2
ProcVersionSignature: Ubuntu 3.16.0-24.32-generic 3.16.4
Uname: Linux 3.16.0-24-generic x86_64
ApportVersion: 2.14.7-0ubuntu8
Architecture: amd64
CurrentDesktop: KDE
Date: Tue Nov 18 15:55:59 2014
EcryptfsInUse: Yes
InstallationDate: Installed on 2014-11-07 (11 days ago)
InstallationMedia: Kubuntu 14.10 "Utopic Unicorn" - Release amd64 (20141022.1)
PackageArchitecture: all
SourcePackage: virt-manager
UpgradeStatus: No upgrade log present (probably fresh install)

[Mark Grocock (mgrocock)]

I've done some more research into this. It appears that the qemu_ga=True argument to _add_var() on line 241 of /usr/share/virt-manager/virtinst/osdict.py (also line 258 affecting Fedora 18 and later) causes qemu-system-x86_64 to be called with an extra argument:

-chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/rhel7.org.qemu.guest_agent.0,server,nowait

I've managed to get this working by creating /var/lib/libvirt/qemu/channel/target with appropriate ownership:

# mkdir -p /var/lib/libvirt/qemu/channel/target
# chown -R libvirt-qemu:kvm /var/lib/libvirt/qemu/channel

and adding the following to the bottom of /etc/apparmor.d/abstractions/libvirt-qemu:

  /var/lib/libvirt/qemu/channel/target/* rw,

(I'm not an apparmor expert, so there may well be a better way of doing this.)

[Jeffrey Bouter (jbouter)]

Taking the steps Mark Grocock posted did not resolve this issue for me. I have no idea where else it may go wrong. The issue remains the same:

Unable to complete install: 'internal error: process exited while connecting to monitor: 2014-12-11T15:32:03.946345Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora21-server.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
2014-12-11T15:32:03.946450Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora21-server.org.qemu.guest_agent.0,server,nowait: chardev: opening backend "socket" failed
'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 91, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1820, in do_install
    guest.start_install(meter=meter)
  File "/usr/share/virt-manager/virtinst/guest.py", line 403, in start_install
    noboot)
  File "/usr/share/virt-manager/virtinst/guest.py", line 467, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3398, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: process exited while connecting to monitor: 2014-12-11T15:32:03.946345Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora21-server.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
2014-12-11T15:32:03.946450Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/fedora21-server.org.qemu.guest_agent.0,server,nowait: chardev: opening backend "socket" failed

[Serge Hallyn (serge-hallyn)]

Thanks - so it looks like virt-aa-helper should be updated to recognize the channels and add a whitelist entry for them.

Do you have xml for a VM with such a channel handy?

[Serge Hallyn (serge-hallyn)]

Actually this bug doesn't appear to be related to apparmoer permissions. The channel is simply not created - /var/lib/libvirt/qemu/channel does not exist, even if qemu-guest-agent package is installed.

[Sebastien Cote (sebas642)]

Same issue here. syslog shows that app-armor is refusing the creation of the socket:

Dec 29 10:07:09: kernel: [ 1957.839479] audit: type=1400 audit(1419876429.922:90): apparmor="DENIED" operation="mknod" profi
le="libvirt-85c4cb3e-a2a1-42ba-af30-5d2d8f989780" name="/var/lib/libvirt/qemu/channel/target/CentOS-7.org.qemu.guest_agent.0" pid=7861 comm
="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=116 ouid=116

Following the steps described above by Mark Grocock fixed the issue.

You need to be careful when editing the app-armor config since the last line of the file closes the 'profile qemu_bridge_helper' group. The added line must be after the curly braces. I have also restarted app-armor, I don't know if it was required.

[Jamie Strandboge (jdstrand)]

While adding this to /etc/apparmor.d/abstractions/libvirt-qemu certainly is a viable workaround:
  /var/lib/libvirt/qemu/channel/target/* rw,

it is not the proper fix because it breaks guest isolation (guests can access other guests target files). Seems like virt-aa-helper should be adjusted to ascertain the name of the 'target' and update /etc/apparmor.d/libvirt/libvirt-<uuid>.files accordingly.

[Serge Hallyn (serge-hallyn)]

Raising priority bc it prevents stocfedora vms from being created using virt-manager

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 1.2.12-0ubuntu9

---------------
libvirt (1.2.12-0ubuntu9) vivid; urgency=medium

  * 9037-virt-aa-helper-add-unix-channels-esp-for-qemu-guest-.patch: Allow
    libvirt domains to start when using qemu guest agent. (LP: #1393842)
 -- Serge Hallyn <email address hidden> Mon, 06 Apr 2015 11:14:03 -0500

[Benjamin Geese (ben-8409)]

Sandly, it seems this is not fixed yet. I have libvirt-1.2.12 and checked my system (vivid) is up to date. I still get the error reported above.

Unable to complete install: 'internal error: process exited while connecting to monitor: 2015-04-09T09:29:32.183316Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/centos7.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: No such file or directory
'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 91, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1819, in do_install
    guest.start_install(meter=meter)
  File "/usr/share/virt-manager/virtinst/guest.py", line 403, in start_install
    noboot)
  File "/usr/share/virt-manager/virtinst/guest.py", line 467, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3422, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: process exited while connecting to monitor: 2015-04-09T09:29:32.183316Z qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/centos7.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: No such file or directory

[Petter Adsen (ducasse)]

Do a "mkdir -p /var/lib/libvirt/qemu/channel/target/" - that should fix it. It should have been created on install, but wasn't, and I guess the package update doesn't create the directory. You might also make sure that owner and group are set correctly, here they are "libvirt-qemu" and "kvm", respectively.

[Serge Hallyn (serge-hallyn)]

Yes, I forgot to have postinst create that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 1.2.12-0ubuntu11

---------------
libvirt (1.2.12-0ubuntu11) vivid; urgency=medium

  * create /var/lib/libvirt/qemu/channel/target (LP: #1393842)
    - libvirt-bin.dirs: add /var/lib/libvirt/qemu/channel/target
    - libvirt-bin.postinst: chown target directory to libvirt-qemu:kvm so
      qemu can create the unix sockets.
 -- Serge Hallyn <email address hidden> Thu, 09 Apr 2015 10:40:05 -0500

[AlexWBaule (alexwbaule)]

The Ubuntu 14.04.3 LTS has this issue too.

Fresh install, today. All updates applied (upgrade and dist-upgrade).

When the update package will be released ?

The lastest version is:

root@ubuntu-kvm:~# dpkg -l | grep libvirt;
ii libvirt-bin 1.2.2-0ubuntu13.1.14 amd64 programs for the libvirt library
iU libvirt0 1.2.2-0ubuntu13.1.14 amd64 library for interfacing with different virtualization systems

First the Directory is not created: (/var/lib/libvirt/qemu/channel/target), creating correctly with the correct owner, has the another issue (about apparmor).

Aug 26 23:34:59 ubuntu-kvm kernel: [ 1198.433256] audit: type=1400 audit(1440642899.122:29): apparmor="DENIED" operation="mknod" profile="libvirt-0fb6b5bc-3a03-4ccb-9950-8eb4e243685e" name="/var/lib/libvirt/qemu/channel/target/rhel6.6.org.qemu.guest_agent.0" pid=3304 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=106 ouid=106

[Serge Hallyn (serge-hallyn)]

Fix is simple enough, I've added it to my list of things to SRU to trusty. I'm hoping to get to it this week.

[AlexWBaule (alexwbaule)]

Hi Serge,

Can you post here the fix ?

So I do the fix on my server until it comes out the package.

Tks.

[AlexWBaule (alexwbaule)]

Hi again Serge, i see the source. Sorry, i think is some configuration file, but is in c source file, need to compile. Ignore my request .

[Serge Hallyn (serge-hallyn)]

I'll push the package for sru today, and post the debdiff here so you
can build your own.

[AlexWBaule (alexwbaule)]

Tks Serge ! i got the diff, rebuild the package and Ok, it's work.

[Brian Murray (brian-murray)]

Hello Mark, or anyone else affected,

Accepted libvirt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.2.2-0ubuntu13.1.15 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Matteo Panella (mpanella)]

Hi Serge,

at first glance the libvirt version in -proposed works when the profile is generated, but virt-aa-helper chokes on profile updates (e.g. media change via virt-manager):

virt-aa-helper: error: /var/lib/libvirt/qemu/channel/target/guineapig.org.qemu.guest_agent.0
virt-aa-helper: error: skipped restricted file
virt-aa-helper: error: invalid VM definition

A simple action like changing cdrom media requires taking down the guest and starting it back up again as soon as a guest agent channel is added, so marking as verification failed.

[The Loeki (the-loeki)]

Confirmed. Update in proposed works as mpanella says

[The Loeki (the-loeki)]

bummer workaround:

for xml in /etc/libvirt/qemu/*.xml; do VM=$(basename $xml|cut -d. -f1); rm /var/lib/libvirt/qemu/channel/target/${VM}.org.qemu.guest_agent.0; virsh dumpxml $VM | /usr/lib/libvirt/virt-aa-helper -c -u libvirt-`virsh domuuid $VM`; virsh start $VM; done

[Serge Hallyn (serge-hallyn)]

Just to be sure - does the same thing happen in wily? That is, is the upstream fix insufficient, or was the SRU missing a piece?

[Matteo Panella (mpanella)]

I don't have a KVM-capable machine with Wily ready at hand, sorry. I'll try to get one ASAP.

[rahul (rhlnair87)]

I am facing the same issue, tried upgrading to 1.2.2-0ubuntu13.1.15. The issue still persists :(

[Serge Hallyn (serge-hallyn)]

Hi,

Using a centos 7 dvd iso and the 'rhel 7 or above' choice when creating a VM in virt-manager,

using the stock trusty image i was able to reproduce this.

Using 1.2.2-0ubuntu13.1.15 from trusty-proposed i was not.

So this *does* solve the issue for me.

[Serge Hallyn (serge-hallyn)]

@rahul,

can you show the error message yo ugot when you tried with the upgraded libvirt?

[Serge Hallyn (serge-hallyn)]

@rahul

ping?

[RussianNeuroMancer (russianneuromancer)]

This issue starting to happen for me after upgrade from Wily to Xenial. Bunch of VMs have org.qemu.guest_agent.0 channel unable to start after upgrade with same error in syslog. On other hand, VMs without org.qemu.guest_agent.0 channel working. As workaround, removing org.qemu.guest_agent.0 channel from all VMs make it works without errors.

[Serge Hallyn (serge-hallyn)]

Thanks - two most likely explanations are that there was a regression
in the apparmor policy, or the filename has changed Could you check
syslog for a related DENIED message in syslog and post it here?

[RussianNeuroMancer (russianneuromancer)]

> Could you check syslog for a related DENIED message in syslog and post it here?

[ 3398.651077] audit: type=1400 audit(1455858424.227:1496): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-aa613ca3-5fff-467e-be3b-7752dc07e856" pid=4326 comm="apparmor_parser"
[ 3398.664393] audit: type=1400 audit(1455858424.243:1497): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="qemu_bridge_helper" pid=4326 comm="apparmor_parser"
[ 3399.035892] audit: type=1400 audit(1455858424.611:1498): apparmor="DENIED" operation="mknod" profile="libvirt-aa613ca3-5fff-467e-be3b-7752dc07e856" name="/var/lib/libvirt/qemu/channel/target/domain-monitoring/org.qemu.guest_agent.0" pid=4328 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=102 ouid=102

> or the filename has changed
I doesn't rename anything in upgrade process. But just to be sure - what I need to check here?

[Serge Hallyn (serge-hallyn)]

Looking at the source, virt-aa-helper should still be doing the right
thing to add an exception for that channel.

For a VM which has that channel, could you post the

/etc/apparmor.d/libvirt/libvirt-<uuid>.files

replacing <uuid> with the vm's uuid, of course.

[RussianNeuroMancer (russianneuromancer)]

~$ cat /etc/apparmor.d/libvirt/libvirt-aa613ca3-5fff-467e-be3b-7752dc07e856.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
  "/var/log/libvirt/**/monitoring.log" w,
  "/var/lib/libvirt/qemu/domain-monitoring/monitor.sock" rw,
  "/var/run/libvirt/**/monitoring.pid" rwk,
  "/run/libvirt/**/monitoring.pid" rwk,
  "/var/run/libvirt/**/*.tunnelmigrate.dest.monitoring" rw,
  "/run/libvirt/**/*.tunnelmigrate.dest.monitoring" rw,
  "/var/lib/libvirt/images/monitoring.img" rw,
  "/dev/net/tun" rw,

[Serge Hallyn (serge-hallyn)]

Confirmed this has regressed in xenial

[Serge Hallyn (serge-hallyn)]

Adding 'capability mknod' to /etc/apparmor.d/abstractions/libvirt-qemu solves it for me.

I'm not sure we want to add that to all VMs. Do we need to add it to the policy during virt-aa-helper?

[Jamie Strandboge (jdstrand)]

I'm not keen on allowing mknod in the general case. It makes a lot of sense to me add it (with comment ideally) via virt-aa-helper.

[Serge Hallyn (serge-hallyn)]

I'm trying:

Index: libvirt/src/security/virt-aa-helper.c
===================================================================
--- libvirt.orig/src/security/virt-aa-helper.c
+++ libvirt/src/security/virt-aa-helper.c
@@ -939,6 +939,14 @@ add_file_path(virDomainDiskDefPtr disk,
 }

 static int
+is_qemu_guest_agent(virDomainChrDefPtr channel)
+{
+
+ return channels->targetType == VIR_DOMAIN_CHR_CHANNEL_TARGET_TYPE_VIRTIO &&
+ STREQ_NULLABLE(channels->target.name, "org.qemu.guest_agent.0";
+}
+
+static int
 get_files(vahControl * ctl)
 {
     virBuffer buf = VIR_BUFFER_INITIALIZER;
@@ -1034,6 +1042,8 @@ get_files(vahControl * ctl)
              ctl->def->channels[i]->source.type == VIR_DOMAIN_CHR_TYPE_PIPE) &&
             ctl->def->channels[i]->source.data.file.path &&
             ctl->def->channels[i]->source.data.file.path[0] != '\0')
+ if (is_qemu_guest_agent(ctl->def->channels[i]))
+ virBufferAsprintf(buf, " capability mknod,\n");
             if (vah_add_file_chardev(&buf,
                                      ctl->def->channels[i]->source.data.file.path,
                                      "rw",

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 1.3.1-1ubuntu6

---------------
libvirt (1.3.1-1ubuntu6) xenial; urgency=medium

  * d/apparmor/libvirt-qemu: generalize the qemu-block-extra libs line.
    (LP: #1554761)
  * d/p/ubuntu/virt-aa-helper-add-mknod-for-guest-agent.patch: add mknod
    capability if there is a qemu guest agent. (LP: #1393842)

 -- Serge Hallyn <email address hidden> Wed, 09 Mar 2016 18:45:08 -0800

[Jamie Strandboge (jdstrand)]

I understand why you are doing this, but this means that a malicious guest is now able to create, for example, a block device with only DAC protecting the host. Since qemu on Ubuntu runs as non-root, this isn't completely horrible, but since apparmor doesn't have fine-grained mediation of mknod, it would be better if the guest agent were modified to use a socket (perhaps abstract?) so the mknod was not required.

[Serge Hallyn (serge-hallyn)]

Quoting Jamie Strandboge (<email address hidden>):
> I understand why you are doing this, but this means that a malicious
> guest is now able to create, for example, a block device with only DAC
> protecting the host. Since qemu on Ubuntu runs as non-root, this isn't
> completely horrible, but since apparmor doesn't have fine-grained
> mediation of mknod, it would be better if the guest agent were modified
> to use a socket (perhaps abstract?) so the mknod was not required.

Agreed that would be better. Do you want to open a bug against the QEMU
project and qemu package to that effect?

[Anthony Kamau (ak-launchpad)]

I've been struggling with this for nearly 2 hours before I realized that I was running a virtualbox vm in headless mode. Was trying to create a qemu-kvm vm and it kept failing with symptoms similar to those reported here.

What would be good is getting qemu-kvm to at least check if another hypervisor is currently running, then allude that fact to the moronic end-user and save the sucker from wasting 2 hours on a Saturday afternoon chasing his tail!

This is the error I was getting:
==========================

[Sat, 26 Mar 2016 15:47:21 virt-manager 490] DEBUG (error:84) error dialog message:
summary=Unable to complete install: 'internal error: process exited while connecting to monitor: ioctl(KVM_CREATE_VM) failed: 16 Device or resource busy
failed to initialize KVM: Device or resource busy
'
details=Unable to complete install: 'internal error: process exited while connecting to monitor: ioctl(KVM_CREATE_VM) failed: 16 Device or resource busy
failed to initialize KVM: Device or resource busy
'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 91, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1819, in do_install
    guest.start_install(meter=meter)
  File "/usr/share/virt-manager/virtinst/guest.py", line 403, in start_install
    noboot)
  File "/usr/share/virt-manager/virtinst/guest.py", line 467, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3497, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: process exited while connecting to monitor: ioctl(KVM_CREATE_VM) failed: 16 Device or resource busy
failed to initialize KVM: Device or resource busy

[Anthony Kamau (ak-launchpad)]

In comparison, when attempting to lunch a VitualBox VM, it fails with a slightly better error message at least directing one to investigate what else would be using a hypervisor. But it also suggests some rather drastic steps to do with recompiling the kernel to remove KVM kernel extension - wow:

==========================
Failed to open a session for the virtual machine core-plus-vm.

VT-x is being used by another hypervisor (VERR_VMX_IN_VMX_ROOT_MODE).

VirtualBox can't operate in VMX root mode. Please disable the KVM kernel extension, recompile your kernel and reboot (VERR_VMX_IN_VMX_ROOT_MODE).

Result Code: NS_ERROR_FAILURE (0x80004005)
Component: ConsoleWrap
Interface: IConsole {872da645-4a9b-1727-bee2-5585105b9eed}

[Serge Hallyn (serge-hallyn)]

Hi,

this should be fixed in libvirt 1.3.1-1ubuntu8. I'm not sure why it didn't get auto-closed.

Please report if this is stlil broken for you.

[Adam Conrad (adconrad)]

Hello Mark, or anyone else affected,

Accepted libvirt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.2.2-0ubuntu13.1.19 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Serge Hallyn (serge-hallyn)]

With current trusty package I got the error popup as predicted.
After upgrading to -proposed and rebooting, it succeeded.

 tag verification-done
 untag verification-needed
 done

[Ankur (ankur22388)]

Hi All, I am using ubuntu 14.04.4 LTS

VERSION="14.04.4 LTS, Trusty Tahr"

root1@root1-HP-Compaq-8100-Elite-CMT-PC:/var/lib/libvirt/qemu$ dpkg -l | grep libvirt
ii libvirt-bin 1.2.2-0ubuntu13.1.17 amd64 programs for the libvirt library
ii libvirt-dev 1.2.2-0ubuntu13.1.17 amd64 development files for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1.17 amd64 library for interfacing with different virtualization systems

libvirt: QEMU Driver error : internal error: process exited while connecting to monitor: qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/f16x86_64.agent,server,nowait: Failed to bind socket: Permission denied

Still I am getting this error, how to fix this?

I tried upgrading the package but its already at newest version

[Kafui Odzangba Dake (odzangba)]

This bug still affects 14.04. Upgrading libvirt to 1.2.2-0ubuntu13.1.19 does not fix it.

$ dpkg -l | grep libvirt
ii libvirt-bin 1.2.2-0ubuntu13.1.19 amd64 programs for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1.19 amd64 library for interfacing with different virtualization systems

$ lsb_release -rd
Description: Ubuntu 14.04.4 LTS
Release: 14.04

$ apt-cache policy libvirt-bin
libvirt-bin:
  Installed: 1.2.2-0ubuntu13.1.19
  Candidate: 1.2.2-0ubuntu13.1.19
  Version table:
 *** 1.2.2-0ubuntu13.1.19 0
        500 http://gh.archive.ubuntu.com/ubuntu/ trusty-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.2.2-0ubuntu13.1.17 0
        500 http://gh.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
     1.2.2-0ubuntu13.1.16 0
        500 http://security.ubuntu.com/ubuntu/ trusty-security/main amd64 Packages
     1.2.2-0ubuntu13 0
        500 http://gh.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

Full error message:

Unable to complete install: 'internal error: process exited while connecting to monitor: qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/centos7.0.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: No such file or directory
qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/centos7.0.org.qemu.guest_agent.0,server,nowait: chardev: opening backend "socket" failed
'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 90, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 2277, in _do_async_install
    guest.start_install(meter=meter)
  File "/usr/share/virt-manager/virtinst/guest.py", line 501, in start_install
    noboot)
  File "/usr/share/virt-manager/virtinst/guest.py", line 416, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3606, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: process exited while connecting to monitor: qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/centos7.0.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: No such file or directory
qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/centos7.0.org.qemu.guest_agent.0,server,nowait: chardev: opening backend "socket" failed

[Serge Hallyn (serge-hallyn)]

Did you reboot the system after the upgrade?

(restarting apparmor should suffice, but since this package fixed it for me I'd like to make sure about whether the core fix failed for you, or just the upgrade experience)

[Kafui Odzangba Dake (odzangba)]

Hi Serge, I rebooted the kvm host after the upgrade and still got the error. I can provide additional info from my kvm host if you need.

[Serge Hallyn (serge-hallyn)]

Hi,

no, thanks, actually since this fix patches virt-aa-helper itself, just creating a new vm after the upgrade should have sufficed. No reboot should have been needed. However trying to start a pre-existing vm that previously failed would not work, as the policy needs to be re-generated.

Looking at your error message, the filename doesn't seem to match what we expect in the patch. The patch uses "domain-$(domain-name)", whereas the log says it was trying to use var/lib/libvirt/qemu/channel/target/centos7.0.org.qemu.guest_agent.0.

Still perplexing that it did work for me. Perhaps adding "/var/lib/libvirt/qemu/channel/target/${domain-name}**" to the whitelist in the same place in the same patch would fix everyone and still be safe.

[Martin Pitt (pitti)]

The version of libvirt in the proposed pocket of Trusty that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

[Jernej Jakob (jjakob)]

Still getting this bug in Trusty.

[Thomas Mayer (thomas303)]

I can confirm this bug for up-to-date xenial (16.04). Note that this is a regression for me, which happened within xenial's updates (it was working a few weeks ago with xenial).

ehler beim Starten der Domain: Kann keine Daten empfangen: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 90, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 126, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 83, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1402, in startup
    self._backend.create()
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1035, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: Kann keine Daten empfangen: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt

[Lior Goikhburg (goikhburg)]

1.2.2-0ubuntu13.1.21 Having this bug.

[Christian Ehrhardt  (paelzer)]

Hi,
thanks for the ping, this brought it to my attention, taking a look now ...

First of all to get Fedora/Virt-manager/... out of scope here a guest without virt-manager or anything else.
Using uvtool to create a very basic guest based on daily cloud images

$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
$ uvt-kvm create --password=ubuntu kvmguest-testgachannel release=xenial arch=amd64 label=daily

In later versions of libvirt/qemu in Xenial for example the apparmor rules automatically get a rule for channels of the guest agent:
   # for qemu guest agent channel
   owner "/var/lib/libvirt/qemu/channel/target/domain-kvmguest-testchannel/**" rw,
The individual guests are namespaced by the domain name (?now?) which makes it easier as there is no rule-per-channel needed as in the past.

But lets try to add a manual channel to see its path and behavior.
The raw basics of a guest channel would be:
    <channel type='unix'>
      <source mode='bind' />
      <target type='virtio' name='org.qemu.guest_agent.0'/>
    </channel>
Adding that (and not more) lets libvirt fill in the extra defaults.

The path that gets auto-assigned does not match the expected guest agent paths above.
That gets expanded on Xenial to:
    <channel type='unix'>
      <source mode='bind'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
But on Trusty it becomes:
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/kvmguest-testgachannel.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

See the path out that is still explicit and also not following the namespaceing of later versions.
The Xenial version generates the path on guest instantiation on the expected path and works as-is.
"-chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-kvmguest-testgachannel/org.qemu.guest_agent.0,server,nowait"

On Trusty this fails as reported:
$ virsh start kvmguest-testgachannel
error: Failed to start domain kvmguest-testgachannel
error: internal error: process exited while connecting to monitor: qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/kvmguest-testgachannel.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: No such file or directory
qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/kvmguest-testgachannel.org.qemu.guest_agent.0,server,nowait: chardev: opening backend "socket" failed

Yet there is no apparmor deny associated, it seems it just fails due to some of the underlying paths being non-existant (maybe it apparmor-fails later once they exist).
Yes that is step 1, after:
$ mkdir /var/lib/libvirt/qemu/channel
$ mkdir /var/lib/libvirt/qemu/channel/target
$ chown libvirt-qemu:kvm /var/lib/libvirt/qemu/channel
$ chown libvirt-qemu:kvm /var/lib/libvirt/qemu/channel/target...

[Christian Ehrhardt  (paelzer)]

There actually is the common virt-aa-helper on channels even back then in Trusty.
This was changed a few times and the special tweak that generates the rule was dropped later as along the new namespacing there are now valid rules per entry.

Anyway for trusty backporting all those complex changes would be not in the SRU mindset, so stick to the proposal I made above.

Please - at least one of the affected users, test the ppa in [1].
If that is successful for you as well and you are willing to also help me verify the eventual SRU we could go forward with that.

My Testing from ppa seems good - log below:

#1 clean env (dir not pre-existing)
#1.1 dir exists after install - ok
#1.2 right ownership - ok
#1.3 socket created - ok
     /var/lib/libvirt/qemu/channel/target/kvmguest-testgachannel.org.qemu.guest_agent.0=
#1.4 apparmor rule - ok
     owner "/var/lib/libvirt/qemu/channel/target/kvmguest-testgachannel.**" rw,
#1.5 Guest working - ok

#2 dir pre-existing but under right ownership/perm
#2.1 - #2.5 as in #1 - ok
#2.6 - no error/conflict due to existing dir

#3 dir pre-existing but under other ownership/perm
#3.1 dir exists after install - ok
#3.2 ownership preserved from before install - ok
#3.3 - apparmor rule creates correctly - ok
#3 fails due to ownership not allowing qemu to create our example guest, but we want to preserve what a user has set up - so ok

[1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2923

[Christian Ehrhardt  (paelzer)]

Waiting for info by users, setting status incomplete for now.

And as a side note #56 is something else or at least worth a new bug to analyze separately. @Thomas Mayer - If you are still affected please open a new bug so we can check out the details of your case.

[Christian Ehrhardt  (paelzer)]

Since it is reproducible and worth to fix I have prepared a MP for an SRU to be reviewed.

[Christian Ehrhardt  (paelzer)]

Added SRU Template in anticipation of the MP review

[Christian Ehrhardt  (paelzer)]

MP Review and tests were good, the package is waiting for SRU Team in trusty-unapproved now.

[Brian Murray (brian-murray)]

Hello Mark, or anyone else affected,

Accepted libvirt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.2.2-0ubuntu13.1.22 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Christian Ehrhardt  (paelzer)]

# left my old dirs as-is (bad setup intentionally) after upgrade
$ dpkg -l libvirt-bin | tee
ii libvirt-bin 1.2.2-0ubuntu13.1.22
$ virsh start kvmguest-testgachannel
testgachannel.org.qemu.guest_agent.0,server,nowait: Failed to bind socket: Permission denied
$ ll /var/lib/libvirt/qemu/
drwxr-xr-x 3 ubuntu ubuntu 4096 Aug 28 11:12 channel/
drwxr-xr-x 2 ubuntu kvm 4096 Aug 28 11:19 target/

# Installs the dirs correctly if not avail (Default case)
$ rm -rf /var/lib/libvirt/qemu/channel
$ apt install --reinstall libvirt-bin
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libfreetype6 os-prober
Use 'apt-get autoremove' to remove them.
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 0 not upgraded.
Need to get 0 B/2070 kB of archives.
After this operation, 0 B of additional disk space will be used.
(Reading database ... 28258 files and directories currently installed.)
Preparing to unpack .../libvirt-bin_1.2.2-0ubuntu13.1.22_amd64.deb ...
libvirt-bin stop/waiting
Unpacking libvirt-bin (1.2.2-0ubuntu13.1.22) over (1.2.2-0ubuntu13.1.22) ...
Processing triggers for libc-bin (2.19-0ubuntu6.13) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up libvirt-bin (1.2.2-0ubuntu13.1.22) ...
libvirt-bin start/running, process 28874
Setting up libvirt-bin dnsmasq configuration.

$ ll /var/lib/libvirt/qemu/channel/target/
total 8
drwxr-xr-x 2 libvirt-qemu kvm 4096 Sep 8 07:23 ./
drwxr-xr-x 3 libvirt-qemu kvm 4096 Sep 8 07:23 ../

# Now starting fine
$ virsh start kvmguest-testgachannel
Domain kvmguest-testgachannel started

# Rule created with namespace
$ grep target /etc/apparmor.d/libvirt/libvirt-4ec6a091-8aef-4bab-b8a4-ca46e91ed18f.files
  owner "/var/lib/libvirt/qemu/channel/target/kvmguest-testgachannel.**" rw,

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 1.2.2-0ubuntu13.1.22

---------------
libvirt (1.2.2-0ubuntu13.1.22) trusty; urgency=medium

  * fix guest channel support (LP: #1393842).
    - d/p/virt-aa-helper-add-trusty-guest-agent-rule.patch: add apparmor rule
      for channels within guest namespace.
    - d/libvirt-bin.postinst: create channel directories if needed.

 -- Christian Ehrhardt <email address hidden> Mon, 28 Aug 2017 12:14:08 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.