kolla-ansible

boot from volume failed

Bug #1670676 reported by Jeffrey Zhang on 2017-03-07

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	kolla-ansible	Fix Released	Critical	Jeffrey Zhang	kolla-ansible pike-2 "pike"
	Ocata	Fix Released	Critical	Jeffrey Zhang	kolla-ansible ocata-rc2 "rc2"

Bug Description

This is related to https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L56,L80

boot from volume failed with following error mesasge

2017-03-07 19:28:58.165 7 INFO nova.virt.libvirt.driver [req-aa7921e6-8d37-4cc3-891b-d032f0dfb298 - - - - -] [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] Deleting instance files /var/lib/nova/instances/c00ec75b-43fc-4a24-a6a0-754496a8fcaf_del
2017-03-07 19:28:58.166 7 INFO nova.virt.libvirt.driver [req-aa7921e6-8d37-4cc3-891b-d032f0dfb298 - - - - -] [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] Deletion of /var/lib/nova/instances/c00ec75b-43fc-4a24-a6a0-754496a8fcaf_del complete
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [req-aa7921e6-8d37-4cc3-891b-d032f0dfb298 - - - - -] [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] Instance failed to spawn
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] Traceback (most recent call last):
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] yield resources
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 1930, in _build_and_run_instance
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] block_device_info=block_device_info)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2688, in spawn
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] destroy_disks_on_failure=True)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5099, in _create_domain_and_network
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] destroy_disks_on_failure)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] self.force_reraise()
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] six.reraise(self.type_, self.value, self.tb)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5071, in _create_domain_and_network
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] post_xml_callback=post_xml_callback)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4989, in _create_domain
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] guest.launch(pause=pause)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 145, in launch
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] self._encoded_xml, errors='ignore')
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] self.force_reraise()
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] six.reraise(self.type_, self.value, self.tb)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 140, in launch
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] return self._domain.createWithFlags(flags)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] rv = execute(f, *args, **kwargs)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] six.reraise(c, e, tb)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] rv = meth(*args, **kwargs)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] File "/var/lib/kolla/venv/lib/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] libvirtError: internal error: qemu unexpectedly closed the monitor: 2017-03-07T11:28:57.777001Z qemu-kvm: -drive file=rbd:volumes/volume-7fbfb8c0-851f-4313-ae2b-89db3e37700a:id=cinder:auth_supported=cephx\;none:mon_host=172.16.103.11\:6789\;172.16.103.12\:6789\;172.16.103.13\:6789,file.password-secret=virtio-disk0-secret0,format=raw,if=none,id=drive-virtio-disk0,serial=7fbfb8c0-851f-4313-ae2b-89db3e37700a,cache=writeback,discard=unmap: error connecting: Operation not permitted
2017-03-07 19:28:58.305 7 ERROR nova.compute.manager [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf]
2017-03-07 19:28:58.307 7 INFO nova.compute.manager [req-aa7921e6-8d37-4cc3-891b-d032f0dfb298 - - - - -] [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] Terminating instance
2017-03-07 19:28:58.319 7 INFO nova.virt.libvirt.driver [-] [instance: c00ec75b-43fc-4a24-a6a0-754496a8fcaf] Instance destroyed successfully.
2017-03-07 19:28:58.353 7 INFO os_vif [req-aa7921e6-8d37-4cc3-891b-d032f0dfb298 - - - - -] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:3b:59:48,bridge_name='qbrc0fed760-a2',has_traffic_filtering=True,id=c0fed760-a2dd-48c7-aa9e-843d0b5959bc,network=Network(5fbeb2ff-9cbf-4318-9b2c-4d8cd03b1c48),plugin='ovs',port_profile=VIFPortProfileBase,preserve_on_delete=False,vif_name='tapc0fed760-a2')

See original description

Jeffrey Zhang (jeffrey4l) on 2017-03-07

Changed in kolla-ansible:
assignee:	nobody → Jeffrey Zhang (jeffrey4l)
milestone:	none → pike-1
importance:	Undecided → Critical

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-07: Fix proposed to kolla-ansible (master)

Fix proposed to branch: master
Review: https://review.openstack.org/442519

Changed in kolla-ansible:
status:	New → In Progress

Jeffrey Zhang (jeffrey4l) on 2017-03-07

description:

updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-08: Fix proposed to kolla-ansible (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/443144

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-08: Fix merged to kolla-ansible (master)

Reviewed: https://review.openstack.org/442519
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=d06efcecc52b00f3e26dce3016accd04f6a52793
Submitter: Jenkins
Branch: master

commit d06efcecc52b00f3e26dce3016accd04f6a52793
Author: Jeffrey Zhang <email address hidden>
Date: Tue Mar 7 22:03:50 2017 +0800

Fix booting from volume failure

    Booting from volume require cinder's ceph client secret now. Move cinder
    before nova in site.yml, because nova depends on cinder ceph client key
    now.

Change-Id: I01c9ed80843d98305b8963894c4917c21a35d3ac
Closes-Bug: #1670676

Changed in kolla-ansible:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-08: Fix merged to kolla-ansible (stable/ocata)

Reviewed: https://review.openstack.org/443144
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=de4fe5311a3aac6562a119eb8f146f117feb0ba8
Submitter: Jenkins
Branch: stable/ocata

commit de4fe5311a3aac6562a119eb8f146f117feb0ba8
Author: Jeffrey Zhang <email address hidden>
Date: Tue Mar 7 22:03:50 2017 +0800

Fix booting from volume failure

    Booting from volume require cinder's ceph client secret now. Move cinder
    before nova in site.yml, because nova depends on cinder ceph client key
    now.

    Change-Id: I01c9ed80843d98305b8963894c4917c21a35d3ac
    Closes-Bug: #1670676
    (cherry picked from commit d06efcecc52b00f3e26dce3016accd04f6a52793)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-03-08: Fix included in openstack/kolla-ansible 4.0.0.0rc2

This issue was fixed in the openstack/kolla-ansible 4.0.0.0rc2 release candidate.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-06-09: Fix included in openstack/kolla-ansible 5.0.0.0b2

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b2 development milestone.

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1671023

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.