Launching VM over RBD backend crashes the compute node

[Dmitry Borodaenko]

Fuel 4.0 ISO #214, non-HA, Ubuntu, controller, 2x compute, all Ceph options enabled (including Ceph RBD backend for ephemeral volumes).

After creating a RAW CirrOS image and launching a VM from it using following commands:

qemu-img convert -O raw /usr/share/cirros-testvm/cirros-0.3.1-x86_64-disk.img /usr/share/cirros-testvm/cirros-0.3.1-x86_64-disk.raw
glance image-create --name cirros-raw --is-public yes --container-format bare --disk-format raw < /usr/share/cirros-testvm/cirros-0.3.1-x86_64-disk.raw
nova boot --flavor 1 --image cirros-raw vm0

the compute node that tries to launch the image locks up, last messages in the logs are:

==> /var/log/daemon.log <==
<30>Dec 30 18:23:42 node-7 ntpd[17142]: Listen normally on 17 br100 10.108.2.5 UDP 123
<30>Dec 30 18:23:42 node-7 ntpd[17142]: Listen normally on 18 br100 fe80::bc84:4dff:fe34:c459 UDP 123
<30>Dec 30 18:23:42 node-7 ntpd[17142]: peers refreshed
<31>Dec 30 18:23:42 node-7 ntpd[17142]: new interface(s) found: waking up resolver
<30>Dec 30 18:23:43 node-7 dnsmasq[8559]: started, version 2.59 cachesize 150
<30>Dec 30 18:23:43 node-7 dnsmasq[8559]: compile time options: IPv6 GNU-getopt DBus i18n DHCP TFTP conntrack IDN
<30>Dec 30 18:23:43 node-7 dnsmasq-dhcp[8559]: DHCP, static leases only on 10.108.2.2, lease time 2m
<30>Dec 30 18:23:43 node-7 dnsmasq[8559]: reading /etc/resolv.conf
<30>Dec 30 18:23:43 node-7 dnsmasq[8559]: using nameserver 10.108.0.2#53
<27>Dec 30 18:23:43 node-7 dnsmasq[8559]: failed to load names from /var/lib/nova/networks/nova-br100.hosts: No such file or directory
<30>Dec 30 18:23:43 node-7 dnsmasq-dhcp[8559]: read /var/lib/nova/networks/nova-br100.conf

==> /var/log/kern.log <==
<5>Dec 30 18:23:44 node-7 kernel: [253594.093269] type=1400 audit(1388427824.967:37): apparmor="DENIED" operation="capable" parent=1 profile="/usr/sbin/libvirtd" pid=26040 comm="libvirtd" pid=26040 comm="libvirtd" capability=29 capname="audit_write"

[Dmitry Borodaenko (angdraug)]

The problem does not occur on CentOS.

[Dmitry Borodaenko (angdraug)]

The problem also occurs on Ubuntu when Ceph backend is enabled for Glance and Cinder, but disabled for Nova.

[Dmitry Borodaenko (angdraug)]

The problem still occurs when AppArmor is disabled. Last thing Nova does is:

<182>Dec 30 22:17:13 node-2 nova-nova.virt.libvirt.firewall INFO: Called setup_basic_filtering in nwfilter
<182>Dec 30 22:17:13 node-2 nova-nova.virt.libvirt.firewall INFO: Ensuring static filters

[Dmitry Borodaenko (angdraug)]

The issue still occurs after upgading QEMU from 1.4.0 to 1.5.2 (host is Ubuntu Raring), but doesn't reproduce in an environment with Linux kernel 3.12 and QEMU 1.7.0 (host is Debian Sid).