failed to launch instance when enabled ceph

Bug #1631984 reported by Jeffrey Zhang
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
kolla
Fix Released
Critical
Jeffrey Zhang
Mitaka
Fix Released
Critical
Jeffrey Zhang

Bug Description

Got following error when enable ceph on master.

2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [req-3253a752-bc25-4dce-a964-527b33207cf3 5d8d1918323d4010a8fc2164dfb00c55 5365f43555864db6af755d8589a935fe - - -] [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] Instance failed to spawn
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] Traceback (most recent call last):
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2078, in _build_resources
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] yield resources
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 1920, in _build_and_run_instance
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] block_device_info=block_device_info)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2583, in spawn
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] post_xml_callback=gen_confdrive)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4823, in _create_domain_and_network
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] post_xml_callback=post_xml_callback)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4752, in _create_domain
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] guest.launch(pause=pause)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 144, in launch
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] self._encoded_xml, errors='ignore')
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] self.force_reraise()
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] six.reraise(self.type_, self.value, self.tb)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 139, in launch
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] return self._domain.createWithFlags(flags)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] rv = execute(f, *args, **kwargs)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] six.reraise(c, e, tb)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] rv = meth(*args, **kwargs)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] File "/var/lib/kolla/venv/lib/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] libvirtError: internal error: process exited while connecting to monitor: 2016-10-10T14:47:55.424673Z qemu-system-x86_64: -drive file=rbd:vms/751db6d0-4417-4827-8e2c-2677ffd5173f_disk:id=nova:key=AQB4o/tX+oWHHhAAL3NypOGKyb+FLStYiQu0xw==:auth_supported=cephx\;none:mon_host=192.168.23.5\:6789\;192.168.23.8\:6789\;192.168.23.26\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=writeback,discard=unmap: could not open disk image rbd:vms/751db6d0-4417-4827-8e2c-2677ffd5173f_disk:id=nova:key=AQB4o/tX+oWHHhAAL3NypOGKyb+FLStYiQu0xw==:auth_supported=cephx\;none:mon_host=192.168.23.5\:6789\;192.168.23.8\:6789\;192.168.23.26\:6789: Unknown protocol
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f]
2016-10-10 22:47:55.611 7 ERROR nova.compute.manager [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f]
2016-10-10 22:47:55.637 7 INFO nova.compute.manager [req-3253a752-bc25-4dce-a964-527b33207cf3 5d8d1918323d4010a8fc2164dfb00c55 5365f43555864db6af755d8589a935fe - - -] [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] Terminating instance
2016-10-10 22:47:55.666 7 INFO nova.virt.libvirt.driver [-] [instance: 751db6d0-4417-4827-8e2c-2677ffd5173f] Instance destroyed successfully.

Changed in kolla:
milestone: none → newton-rc2
Steven Dake (sdake)
Changed in kolla:
status: New → Confirmed
importance: Undecided → Critical
status: Confirmed → Triaged
Revision history for this message
Jeffrey Zhang (jeffrey4l) wrote :

kolla mitaka

(nova-libvirt)[root@control1 /]# ldd /usr/libexec/qemu-kvm | grep rbd
 librbd.so.1 => /lib64/librbd.so.1 (0x00007f265e4b4000)

kolla master

(nova-libvirt)[root@a1 /]# ldd /usr/bin/qemu-system-x86_64 | grep -i rbd

we upgrade the qemu-kvm to qemu-kvm-ev. seem the qemu-kvm-ev id not compiled with rbd support.

Revision history for this message
Alan Pevec (apevec) wrote :

Please provide RPM versions in both cases:

rpm -qf /usr/libexec/qemu-kvm
rpm -qf /usr/bin/qemu-system-x86_64

Not sure what you have in "kolla master" but /usr/bin/qemu-system-x86_64 is not provided by qemu-kvm-ev-2.3.0-31.el7.16.1.x86_64 please use qemu-kvm.

Revision history for this message
Alan Pevec (apevec) wrote :

With current CentOS Virt SIG repo I get:

# rpm -qf /usr/libexec/qemu-kvm
qemu-kvm-ev-2.3.0-31.el7.16.1.x86_64
# ldd /usr/libexec/qemu-kvm|grep rbd
librbd.so.1 => /lib64/librbd.so.1 (0x00007fbb44be1000)

Revision history for this message
Steven Dake (sdake) wrote :

Thanks Alan. Jeffrey identified the root cause (we were installing qemu-system-x86 instead of qemu-kvm). In the past installing qemu-kvm installed a million dependencies and qemu-system-x86 only installed what was necessary.

We are unable to remove epel as a remo because cinder-volume installs an iscsi package which is required by third party storage vendors.

Regards
-steve

Changed in kolla:
assignee: nobody → Jeffrey Zhang (jeffrey4l)
status: Triaged → Confirmed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to kolla (master)

Reviewed: https://review.openstack.org/385102
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=255198e742e0a09923e4dffcbbe6bd96b10c93be
Submitter: Jenkins
Branch: master

commit 255198e742e0a09923e4dffcbbe6bd96b10c93be
Author: Jeffrey Zhang <email address hidden>
Date: Wed Oct 12 01:28:17 2016 +0800

    Fix the failure to launch instance when enabled Ceph

    We need install qemu-kvm package which come from the Cloud SIG rather
    then qemu-system-x86 which come from EPEL.

    Change-Id: Ibd027069771552d3db0e4fe0c7406ddd1bfc0673
    Closes-Bug: #1631984

Changed in kolla:
status: Confirmed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/kolla 3.0.0.0rc2

This issue was fixed in the openstack/kolla 3.0.0.0rc2 release candidate.

Revision history for this message
Jeffrey Zhang (jeffrey4l) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.