failed to launch instance when enabled ceph

[Jeffrey Zhang]

Got following error when enable ceph on master.

2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [req-3253a752-bc25-4dce-a964-527b33207cf3 5d8d1918323d4010a8fc2164dfb00c55 5365f43555864db6af755d8589a935fe - - -] [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] Instance failed to spawn
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] Traceback (most recent call last):
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2078, in _build_resources
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] yield resources
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 1920, in _build_and_run_instance
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] block_device_info=block_device_info)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2583, in spawn
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] post_xml_callback=gen_confdrive)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4823, in _create_domain_and_network
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] post_xml_callback=post_xml_callback)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4752, in _create_domain
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] guest.launch(pause=pause)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] self._encoded_xml, errors='ignore')
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] self.force_reraise()
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] six.reraise(self.type_, self.value, self.tb)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] return self._domain.createWithFlags(flags)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] rv = execute(f, *args, **kwargs)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] six.reraise(c, e, tb)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] rv = meth(*args, **kwargs)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] libvirtError: internal error: process exited while connecting to monitor: 2016-10-10T14:47:55.424673Z qemu-system-x86_64: -drive file=rbd:vms/751db6d0-4417-4827-8e2c-2677ffd5173f_disk:id=nova:key=AQB4o/tX+oWHHhAAL3NypOGKyb+FLStYiQu0xw==:auth_supported=cephx\;none:mon_host=192.168.23.5\:6789\;192.168.23.8\:6789\;192.168.23.26\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,discard=unmap: could not open disk image rbd:vms/751db6d0-4417-4827-8e2c-2677ffd5173f_disk:id=nova:key=AQB4o/tX+oWHHhAAL3NypOGKyb+FLStYiQu0xw==:auth_supported=cephx\;none:mon_host=192.168.23.5\:6789\;192.168.23.8\:6789\;192.168.23.26\:6789: Unknown protocol
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f]
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f]
2016-10-10 22:47:55.637 7 INFO nova.compute.manager [req-3253a752-bc25-4dce-a964-527b33207cf3 5d8d1918323d4010a8fc2164dfb00c55 5365f43555864db6af755d8589a935fe - - -] [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] Terminating instance
2016-10-10 22:47:55.666 7 INFO nova.virt.libvirt.driver [-] [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] Instance destroyed successfully.

[Jeffrey Zhang (jeffrey4l)]

kolla mitaka

(nova-libvirt)[root@control1 /]# ldd /usr/libexec/qemu-kvm | grep rbd
 librbd.so.1 => /lib64/librbd.so.1 (0x00007f265e4b4000)

kolla master

(nova-libvirt)[root@a1 /]# ldd /usr/bin/qemu-system-x86_64 | grep -i rbd

we upgrade the qemu-kvm to qemu-kvm-ev. seem the qemu-kvm-ev id not compiled with rbd support.

[Alan Pevec (apevec)]

Please provide RPM versions in both cases:

rpm -qf /usr/libexec/qemu-kvm
rpm -qf /usr/bin/qemu-system-x86_64

Not sure what you have in "kolla master" but /usr/bin/qemu-system-x86_64 is not provided by qemu-kvm-ev-2.3.0-31.el7.16.1.x86_64 please use qemu-kvm.

[Alan Pevec (apevec)]

With current CentOS Virt SIG repo I get:

# rpm -qf /usr/libexec/qemu-kvm
qemu-kvm-ev-2.3.0-31.el7.16.1.x86_64
# ldd /usr/libexec/qemu-kvm|grep rbd
librbd.so.1 => /lib64/librbd.so.1 (0x00007fbb44be1000)

[Steven Dake (sdake)]

Thanks Alan. Jeffrey identified the root cause (we were installing qemu-system-x86 instead of qemu-kvm). In the past installing qemu-kvm installed a million dependencies and qemu-system-x86 only installed what was necessary.

We are unable to remove epel as a remo because cinder-volume installs an iscsi package which is required by third party storage vendors.

Regards
-steve

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/385102
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=255198e742e0a09923e4dffcbbe6bd96b10c93be
Submitter: Jenkins
Branch: master

commit 255198e742e0a09923e4dffcbbe6bd96b10c93be
Author: Jeffrey Zhang <email address hidden>
Date: Wed Oct 12 01:28:17 2016 +0800

    Fix the failure to launch instance when enabled Ceph

    We need install qemu-kvm package which come from the Cloud SIG rather
    then qemu-system-x86 which come from EPEL.

    Change-Id: Ibd027069771552d3db0e4fe0c7406ddd1bfc0673
    Closes-Bug: #1631984

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla 3.0.0.0rc2 release candidate.

[Jeffrey Zhang (jeffrey4l)]

mitaka branch is fixed by https://review.openstack.org/#/c/409513/