instance fails to boot with qemu guest agent set in image metadata

Bug #1227912 reported by Ravi Chunduru on 2013-09-19
44
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Undecided
Unassigned
Ubuntu
Undecided
Unassigned

Bug Description

I set the image metadata to enable qemu guest agent. When I bring up instance it fails to open socket due to permission.
I observed this with devstack.

Here is more info

nova image-meta cirros-0.3.1-x86_64-uec set hw_qemu_guest_agent=yes

2013-09-19 15:21:55.717 ERROR nova.compute.manager [req-be5e6b88-77f8-47d5-974e-85e1f875608a demo demo] [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] Error: internal error process exited while connecting to monitor: char device redirected to /dev/pts/27 (label charserial1)
qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-0000000b.sock,server,nowait: Failed to bind socket: Permission denied
chardev: opening backend "socket" failed

2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] Traceback (most recent call last):
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/opt/stack/nova/nova/compute/manager.py", line 1038, in _build_instance
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] set_access_ip=set_access_ip)
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/opt/stack/nova/nova/compute/manager.py", line 1411, in _spawn
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] LOG.exception(_('Instance failed to spawn'), instance=instance)
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/opt/stack/nova/nova/compute/manager.py", line 1408, in _spawn
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] block_device_info)
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2071, in spawn
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] block_device_info, context=context)
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3214, in _create_domain_and_network
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4]
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3157, in _create_domain
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4]
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3152, in _create_domain
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] except Exception as e:
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] rv = execute(f,*args,**kwargs)
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] rv = meth(*args,**kwargs)
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] libvirtError: internal error process exited while connecting to monitor: char device redirected to /dev/pts/27 (label charserial1)
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-0000000b.sock,server,nowait: Failed to bind socket: Permission denied
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4] chardev: opening backend "socket" failed
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4]
2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance: fccd153e-b8f7-4b09-aa25-b531fd618eb4]

Ravi Chunduru (ravivsn) on 2013-09-19
Changed in nova:
importance: Undecided → High
Russell Bryant (russellb) wrote :

Does cirros have the qemu guest agent installed?

Changed in nova:
status: New → Incomplete
tags: added: libvirt

Could be that apparmor is enabled in your host.
Try disabling it.
Or change the apparmor profile to enable the operation you are doing.

Srini

Ravi Chunduru (ravivsn) wrote :

Srini, Thanks for the comment. It came out to be indeed apparmor issue.
I set the following in /etc/libvirt/qemu.conf and VMs came up fine.
security_driver = "none"

Now the question is there must be a way for devstack to set qemu.conf if user needs to use virtio channels or nova to address apparmor template.

Changed in nova:
status: Incomplete → New
wangpan (hzwangpan) wrote :

Hi Ravi, I also got this issue when I test my codes under devstack, and this is the comment about this issue:
https://review.openstack.org/#/c/36372/6..7/etc/nova/nova.conf.sample
The original method to fix this issue is adding a config item for storing the socket file on the host, but Daniel doesn't like this, and I think the better way to notify the user about this trap is adding some doc to the release notes.

Download full text (6.8 KiB)

Hi Wangpan, the exact config settings of the added feature did not come up
on devstack. I tried with virt-inst to make sure its not devstack thing but
still I got permission error. I have to fix libvirt-qemu apparmor settings
to make the VM boot up.

I suggest community to consider the following. I will propose BP on similar
lines
Source path, target name must be configurable to allow other use cases like
access to appliance VMs from host so that configuration can be pushed and
run time status can be fetched.
Fix libvirt apparmor issues or document the settings.

On Wed, Sep 25, 2013 at 4:03 AM, wangpan <email address hidden> wrote:

> Hi Ravi, I also got this issue when I test my codes under devstack, and
> this is the comment about this issue:
> https://review.openstack.org/#/c/36372/6..7/etc/nova/nova.conf.sample
> The original method to fix this issue is adding a config item for storing
> the socket file on the host, but Daniel doesn't like this, and I think the
> better way to notify the user about this trap is adding some doc to the
> release notes.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1227912
>
> Title:
> instance fails to boot with qemu guest agent set in image metadata
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> I set the image metadata to enable qemu guest agent. When I bring up
> instance it fails to open socket due to permission.
> I observed this with devstack.
>
> Here is more info
>
> nova image-meta cirros-0.3.1-x86_64-uec set hw_qemu_guest_agent=yes
>
> 2013-09-19 15:21:55.717 ERROR nova.compute.manager
> [req-be5e6b88-77f8-47d5-974e-85e1f875608a demo demo] [instance:
> fccd153e-b8f7-4b09-aa25-b531fd618eb4] Error: internal error process exited
> while connecting to monitor: char device redirected to /dev/pts/27 (label
> charserial1)
> qemu-system-x86_64: -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/org.qemu.guest_agent.0.instance-0000000b.sock,server,nowait:
> Failed to bind socket: Permission denied
> chardev: opening backend "socket" failed
>
> 2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance:
> fccd153e-b8f7-4b09-aa25-b531fd618eb4] Traceback (most recent call last):
> 2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance:
> fccd153e-b8f7-4b09-aa25-b531fd618eb4] File
> "/opt/stack/nova/nova/compute/manager.py", line 1038, in _build_instance
> 2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance:
> fccd153e-b8f7-4b09-aa25-b531fd618eb4] set_access_ip=set_access_ip)
> 2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance:
> fccd153e-b8f7-4b09-aa25-b531fd618eb4] File
> "/opt/stack/nova/nova/compute/manager.py", line 1411, in _spawn
> 2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance:
> fccd153e-b8f7-4b09-aa25-b531fd618eb4] LOG.exception(_('Instance failed
> to spawn'), instance=instance)
> 2013-09-19 15:21:55.717 TRACE nova.compute.manager [instance:
> fccd153e-b8f7-4b09-aa25-b531fd618eb4] File
> "/opt/stack/nova/nova/compute/manager.py", line 1408, in _spawn
> 2013-09-19 15:21:55.717 TRACE nova.compu...

Read more...

Changed in nova:
status: New → Invalid
importance: High → Undecided
MNLipp (mnl) wrote :

I have just encounter this problem in Ubuntu 14.04.1. Here's a workaround:

# cd /var/lib/libvirtd/qemu
# mkdir -p channel/target
# chown -R libvirt-qemu:kvm channel/

(The path above is used by libvirt-manager when you create the channel.)

In /etc/apparmor.d/abstractions/libvirt-qemu at the end add:

"/var/lib/libvirt/**/*.org.qemu.guest_agent.0" rwk,

(Reload apparmor profiles).

The line in libvirt-qemu could be generated in the domain specific file by virt-aa-helper to exactly match the name of the domain, but I cannot see a high security risk in being a bit unspecific here (allows one qemu to access the socket of another qemu).

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in ubuntu:
status: New → Confirmed
Luis Mondesi (lemsx1) wrote :

MNLipp workaround moved me from the previous error to this one:

Unable to complete install: 'internal error: Process exited prior to exec: libvirt: error : unable to set AppArmor profile 'libvirt-9a3f213c-b02e-45a1-8791-109f6300d200' for '/usr/bin/qemu-system-x86_64': No such file or directory
'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 91, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 1820, in do_install
    guest.start_install(meter=meter)
  File "/usr/share/virt-manager/virtinst/guest.py", line 403, in start_install
    noboot)
  File "/usr/share/virt-manager/virtinst/guest.py", line 467, in _create_guest
    dom = self.conn.createLinux(start_xml or final_xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 3398, in createLinux
    if ret is None:raise libvirtError('virDomainCreateLinux() failed', conn=self)
libvirtError: internal error: Process exited prior to exec: libvirt: error : unable to set AppArmor profile 'libvirt-9a3f213c-b02e-45a1-8791-109f6300d200' for '/usr/bin/qemu-system-x86_64': No such file or directory

System details:

$> dpkg -S /usr/bin/qemu-system-x86_64
qemu-system-x86: /usr/bin/qemu-system-x86_64

$> COLUMNS=100 dpkg -l libvirt-bin|tail -1

ii libvirt-bin 1.2.8-0ubuntu1 amd64 programs for the libvirt library

$> lsb_release -a

Setting "security_driver=none" and restarting libvirt-bin service at least allowed me to continue.

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.10
Release: 14.10
Codename: utopic

I had the same issue when AppArmor is active (default)
and when I try to enable qemu guest agent inside your
guest :

virsh start test
error: Failed to start domain test
error: internal error: process exited while connecting to monitor: qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/test-virtio.sock,server,nowait: Failed to bind socket: Permission denied
qemu-system-x86_64: -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/test-virtio.sock,server,nowait: chardev: opening backend "socket" failed

audit: type=1400 audit(...): apparmor="DENIED" operation="mknod" profile="libvirt-74c30212-4631-4498-a684-c62db8b2dc21" name="/var/lib/libvirt/qemu/test-virtio.sock" pid=10291 comm="qemu-system-x86" requested_mask="c" denied_mask="c" fsuid=106 ouid=106

    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback' discard='unmap'/>
      <source dev='/var/lib/libvirt/images/test.raw'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
    </controller>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/test-virtio.sock'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>

I am using qemu guest agent to be able to call guest-fstrim
virsh qemu-agent-command <domain> '{"execute":"guest-fstrim"}'
guest-fstrim is doing fstrim on all partitions if discard has been
enabled with virtio-scsi (free up deleted blocks)

Is it possible to apply this patch to libvirt-bin package and allow to write to /var/lib/libvirt/qemu/ ?

For me it makes sense because disabling AppArmor for OpenStack is not a good idea
so you will be exposed to security issues like Venom
https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/VENOM
<< Additionally, the QEMU process is confined by an AppArmor profile that significantly lessens the impact of a vulnerability such as VENOM by reducing the host environment's attack surface >>

$ dpkg -S /etc/apparmor.d/abstractions/libvirt-qemu
libvirt-bin: /etc/apparmor.d/abstractions/libvirt-qemu

The attachment "Fix for" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Patches