VMs won't start after purging apparmor

Bug #585964 reported by Hovik Manucharyan
20
This bug affects 4 people
Affects Status Importance Assigned to Milestone
libvirt (Ubuntu)
Fix Released
Low
Jamie Strandboge

Bug Description

Background on this issue: http://open.eucalyptus.com/forum/libvirt-operation-failed-failed-retrieve-chardev-info-qemu-info-chardev

Whenever I try to run VM instance using virsh, I get this error:

error: monitor socket did not show up.: Connection refused

Error is always reproducible:

==================

root@srv-uec-qa-node02:/var/lib/eucalyptus/instances/admin/i-46D20834# ls -al
total 786432
drwxr-xr-x 2 eucalyptus eucalyptus 4096 2010-05-26 21:49 .
drwxr-xr-x 3 eucalyptus eucalyptus 4096 2010-05-26 20:15 ..
-rw-r--r-- 1 eucalyptus eucalyptus 16153313280 2010-05-26 20:15 disk
-rw------- 1 eucalyptus eucalyptus 571064 2010-05-26 20:15 instance-checkpoint
-rw-r--r-- 1 eucalyptus eucalyptus 3934784 2010-05-24 00:16 kernel
-rw-r--r-- 1 eucalyptus eucalyptus 943 2010-05-26 21:49 libvirt.xml
-rw-r--r-- 1 eucalyptus eucalyptus 3863052 2010-05-24 00:16 ramdisk
root@srv-uec-qa-node02:/var/lib/eucalyptus/instances/admin/i-46D20834# virsh define libvirt.xml
Domain i-46D20834 defined from libvirt.xml

root@srv-uec-qa-node02:/var/lib/eucalyptus/instances/admin/i-46D20834# virsh start i-46D20834
error: Failed to start domain i-46D20834
error: monitor socket did not show up.: Connection refused

root@srv-uec-qa-node02:/var/lib/eucalyptus/instances/admin/i-46D20834#

===============

virsh debug logs: http://slexy.org/view/s21euMfVqK

contents of libvirt.xml: http://slexy.org/view/s20fCTVqzV

strace of libvirtd while I run the command: http://slexy.org/raw/s2I1ZYzm8k

===============
System Info

0. Ubuntu 10.04 Server
1. apparmor packages have been purged
2. selinux/apparmor disabled by using "apparmor=0 selinux=0" as kernel boot parameters
3. Contents of /etc/libvirt/qemu.conf: http://slexy.org/raw/s2hIyg3Dnf
4. Contents of /etc/libvirt/libvirtd.conf: http://slexy.org/raw/s21cfGRjCd
5. Currently installed packages: http://slexy.org/raw/s2Gi4KDogx

Tags: apparmor
Revision history for this message
Hovik Manucharyan (hovik) wrote :
Revision history for this message
Hovik Manucharyan (hovik) wrote :
Revision history for this message
Hovik Manucharyan (hovik) wrote :
Revision history for this message
Hovik Manucharyan (hovik) wrote :
Revision history for this message
Hovik Manucharyan (hovik) wrote :
Revision history for this message
Hovik Manucharyan (hovik) wrote :
Revision history for this message
Hovik Manucharyan (hovik) wrote :

I was able to reproduce this issue again for the 3rd time on a clean install. I believe that anybody can reproduce this with the following steps:

STEP 1: Do a clean Ubuntu Server 10.04 installation. When I installed, I selected "OpenSSH server" as the only additional installation package.

STEP 2: Install libvirt-bin package.

I installed it as such: apt-get install eucalyptus-nc

This downloaded libvirt-bin as a dependency.

After I installed Eucalyptus NC and configured the node properly I verified that this node was able to execute my images properly. Everything worked fine. As part of configuring Eucalyptus-NC, I also created a bridge br0 to my eth0 dev.

STEP 3: Edit /etc/libvirt/qemu.conf

Config change:

security_driver = "none"

I restarted libvirt-bin and verified again that everything was working fine.

This step may have a benign effect or it may act in combination with STEP 4 to cause the issue.

STEP 4: Purge apparmor package & reboot

root@srv-uec-qa-node01:~# apt-get purge apparmor
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
  apparmor* apparmor-utils*
0 upgraded, 0 newly installed, 2 to remove and 9 not upgraded.
After this operation, 4,067kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 44060 files and directories currently installed.)
Removing apparmor-utils ...
Purging configuration files for apparmor-utils ...
Removing apparmor ...
 * Unloading AppArmor profiles [ OK ]
Purging configuration files for apparmor ...
dpkg: warning: while removing apparmor, directory '/etc/apparmor.d/cache' not empty so not removed.
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot

----

Then I reboot.

=============================================================================

After STEP4 my problem is reproduced. I get the following error when I try to run vm instances using libvirtd:

root@srv-uec-qa-node01:/var/lib/eucalyptus/instances/admin/i-5AF109A2# virsh start i-5AF109A2
error: Failed to start domain i-5AF109A2
error: monitor socket did not show up.: Connection refused

Thus, the problem is either STEP4 by itself or combination of STEP 3 + STEP 4.

------

Even if I don't purge apparmor but disable it with a kernel param (apparmor=0), the problem repeats itself, suggesting libvirtd relies on the presence of apparmor in order to function properly.

------

I haven't had 100% success fixing libvirt after this.

In one case, reinstalling apparmor seemed to make libvirtd work again.

However, in another case I wasn't successful. On this system I had disabled apparmor using a kernel boot param. Additionally, this system was in a "broken" state for several days and may have seen other changes beside the boot param. I undid my modifications to the kernel boot parameters, rebooted, reinstalled apparmor, but the problem persisted.

Revision history for this message
Chuck Short (zulcss) wrote :

Why are you removing apparmor?

chuck

Changed in libvirt (Ubuntu):
importance: Undecided → Low
status: New → Incomplete
tags: added: apparmor
Revision history for this message
H.M. (cybrocop) wrote :

I now have a clean install of UEC with apparmor working but it is a problem if someone wants to remove it for whatever reason. This was a test system and I was experimenting and I had assumed that UEC minus apparmor was a supported configuration. Also, I wanted to have the ability to have symlinks from /var/lib/eucalyptus/instances to my own location.

Is apparmor mandatory for use with libvirtd/eucalyptus?

Hovanes

Revision history for this message
H.M. (cybrocop) wrote :

Why is the status of the bug incomplete?

Are you unable to reproduce with the steps I mentioned?

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Hovannes,

While recommended, the use of libvirt without AppArmor is supported. I did not yet have time to look into this, but will get to it this coming week. Assigning back to me and back to NEW until I can look at it.

Changed in libvirt (Ubuntu):
assignee: nobody → Jamie Strandboge (jdstrand)
status: Incomplete → New
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Revised test case:
1. install libvirt-bin and test that a VM can start
2. apt-get purge apparmor
3. reboot
4. try to start a VM

WORKAROUND: simply disabling apparmor via 'security=""' with or without purging seems to work fine. I also noticed that subsequent reboots seemed to work ok (which is a little weird).

Unless you have a specific reason to do so, I *highly* recommend using AppArmor with libvirt. If you really need to disable AppArmor with libvirt, I suggest only disabling the profile instead of all of AppArmor. AppArmor protects several applications in the default install. If you only want to disable AppArmor for libvirt, adjust qemu.conf to use 'security = none' or do: 'sudo touch /etc/apparmor.d/disable/usr.sbin.libvirtd'.

Changed in libvirt (Ubuntu):
status: New → Confirmed
Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Actually, I misspoke. Setting 'security_driver = "none"' does not work due to bug #588369 (likely because of the new stacked security driver system in newer libvirts). However, the following is confirmed to work as expected, disabling AppArmor for only libvirt:

$ sudo touch /etc/apparmor.d/disable/usr.sbin.libvirtd
$ sudo reboot

summary: - Libvirtd -- error: monitor socket did not show up.: Connection refused
+ VMs won't start after purging apparmor
Revision history for this message
Henrik Svensson (henrik-svensson) wrote :

Hi!

I got a similar issue on our production environment...
I noticed that after a clean install shutdown and start of domains works BUT
if you restart the libvirt-bin service you can shutdown domains but not start them again.
If you while in this state purge the libvirt-bin packaga and install it again (with aptitude)
it works until the next restart of libvirt-bin.
In this case I did not touch apparmor and I did not reboiot the server.

Output from the terminal:

--------------------------------------------------------------------
root@srvsxu0001:~# virsh
Welcome to virsh, the virtualization interactive terminal.

Type: 'help' for help with commands
       'quit' to quit

virsh # list
 Id Name State
----------------------------------
  1 xus1004-001 running

virsh # shutdown xus1004-001
Domain xus1004-001 is being shutdown

virsh # list
 Id Name State
----------------------------------

virsh # start xus1004-001
Domain xus1004-001 started

virsh # list
 Id Name State
----------------------------------
  2 xus1004-001 running

virsh # quit

root@srvsxu0001:~# service libvirt-bin stop
libvirt-bin stop/waiting

root@srvsxu0001:~# service libvirt-bin start
libvirt-bin start/running, process 13645

root@srvsxu0001:~# virsh
Welcome to virsh, the virtualization interactive terminal.

Type: 'help' for help with commands
       'quit' to quit

virsh # list
 Id Name State
----------------------------------
  2 xus1004-001 running

virsh # shutdown xus1004-001
Domain xus1004-001 is being shutdown

virsh # list
 Id Name State
----------------------------------

virsh # start xus1004-001
error: Failed to start domain xus1004-001
error: monitor socket did not show up.: No such file or directory

virsh #
--------------------------------------------------------------------

Revision history for this message
Jamie Strandboge (jdstrand) wrote :

Henrik, yours is a different issue. Can you please file a new bug?

Revision history for this message
Henrik Svensson (henrik-svensson) wrote :

Ok.

After extensive testing I found out that this had to do with the vnc_tls parameter in qemu.conf.
If it is set to 1 it fails with the error message above.

I will investigate more and file a bug if its not just misconfiguration.
Thanks for your fast reply.

Revision history for this message
kuschky (kakuschky) wrote :

Hello, after last Ubuntu update I can't start my KVM machines. So I did a clean basic installation and install only libvirt-bin and kvm.

uname -a : Linux server1 2.6.32-23-server #37-Ubuntu SMP Fri Jun 11 09:11:11 UTC 2010 x86_64 GNU/Linux

But the error was still there.

I followed the hints in this thread and did the following:

- in qemu.conf I set security_driver = "none"

- I did purge appamor "apt-get purge apparmor"

- In qemu.conf I set vnc_tls = 0

- reboot the system

But also with this modifications I still got the same error:

error: Failed to start domain testsystem.mynet.de
error: monitor socket did not show up.: Connection refused

syslog show the following

Jul 28 08:09:13 eq4 libvirtd: 08:09:13.456: error : qemuMonitorOpenUnix:268 : monitor socket did not show up.: Connection refused
Jul 28 08:09:13 eq4 libvirtd: 08:09:13.456: error : qemuConnectMonitor:822 : Failed to connect monitor for testsystem.mynet.de#012

Is there any other solution to get kvm running than to use another distribution?

Regards Michael

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (4.7 KiB)

This bug was fixed in the package libvirt - 0.8.3-1ubuntu1

---------------
libvirt (0.8.3-1ubuntu1) maverick; urgency=low

  * Merge from debian unstable with security fixes
  * Fixes:
    - LP: #588369
    - LP: #585964
  * Remaining changes:
    - debian/control:
      + Build-Depends on qemu-kvm, not qemu
      + Build-Depends on open-iscsi-utils, not open-iscsi
      + Build-Depends on libxml2-utils
      + Build-Depends on libapparmor-dev and Suggests apparmor
      + Bump bridge-utils, dnsmasq-base, netcat-openbsd, and iptables
        to Depends of libvirt-bin
      + Drop lvm2, qemu-kvm and qemu to Suggests
      + We call libxen-dev libxen3-dev, so change all references
      + Rename Vcs-* to XS-Debian-Vcs-*
    - debian/libvirt-bin.postinst:
      + rename the libvirt group to libvirtd
      + add each admin user to the libvirtd group
      + reload apparmor profiles
    - debian/libvirt-bin.postrm:
      + rename the libvirt group to libvirtd
      + remove apparmor symlinks on purge
    - debian/README.Debian: add AppArmor section based on the upstream
      documentation
    - debian/rules:
      + update DEB_DH_INSTALLINIT_ARGS for upstart
      + add DEB_MAKE_CHECK_TARGET := check
      + use --with-apparmor
      + copy apparmor and apport hook to debian/tmp
    - add debian/libvirt-bin.upstart
    - debian/libvirt-bin.dirs: add /etc/apparmor.d/abstractions,
      /etc/apparmor.d/disable, /etc/apparmor.d/force-complain,
      /etc/apparmor.d/libvirt, /etc/cron.daily and
      /usr/share/apport/package-hooks
    - add debian/libvirt-bin.cron.daily
    - add debian/libvirt-bin.apport
    - debian/libvirt-bin.install: install apparmor profiles, abstractions
      and apport hook
    - debian/apparmor:
      - add TEMPLATE
      - add libvirt-qemu abstraction
      - add usr.lib.libvirt.virt-aa-helper
      - add usr.sbin.libvirtd
    - debian/patches/series:
      + don't apply 0002-qemu-disable-network.diff.patch
      + don't apply 0005-Terminate-nc-on-EOF.patch. Use
        9009-autodetect-nc-params.patch instead
      + 9000-delayed_iff_up_bridge.patch (refreshed)
      + 9001-dont_clobber_existing_bridges.patch
      + 9002-better_default_uri_virsh.patch (refreshed)
      + 9003-better-default-arch.patch (refreshsed)
      + 9004-libvirtd-group-name.patch
      + 9005-increase-unix-socket-timeout.patch (refreshed)
      + 9006-default-config-test-case.patch
      + 9007-fix-daemon-conf-ftbfs.patch (updated)
      + 9008-run-as-root-by-default.patch (refreshed)
      + 9009-autodetect-nc-params.patch (refreshed)
      + 9010-dont-disable-ipv6.patch (refreshsed)
      + 9011-move-ebtables-script.patch (refreshed)
  * Dropped the following patches included/fixed upstream:
    - 9012-fix-nodeinfotest-ftbfs.patch
    - 9013-apparmor-lp457716.patch
  * Disable virtualbox support since virtualbox-ose is not in main
    - debian/control: remove virtualbox-ose build dependency
    - debian/rules: use --without-vbox
  * debian/patches/9012-apparmor-dont-ignore-open.patch: fix logic when
    using virDomainDiskDefForeachPath() and add tests. This can be removed
    in 0.8.4.
  * debian/apparmor/usr.sbin.libvirtd: add capability fseti...

Read more...

Changed in libvirt (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.