| 2017-05-18 20:10:24 |
Piyush Srivastava |
description |
Description
===========
Nova doesn't cleanup port when instance launch fails on a hypervisor due to virt issues, instance is automatically rescheduled to a different hypervisor which results in duplicate ports for the instance.
Steps to reproduce
==================
A chronological list of steps which will bring off the
issue you noticed:
* I booted a VM (openstack server create --image cirros --flavor m1.tiny --nic net-id=network-id test-vm)
* then instance for scheduled on hypervisor h1
* then instance launch failed due to virt errors on h1 (virt issues on the h1)
* then nova did not cleanup the port while cleaning up the failed instance
* then instance was rescheduled on hypervisor h2
* then instance booted successfully
* then I did openstack server list
* then instance showed up with two IPs
* then I did openstack port list
* then there were two ports for the same instance
Expected result
===============
There should only be one port for an instance
Actual result
=============
There were two ports for the instance
Environment
===========
1. Exact version of OpenStack you are running.
openstack-nova-common.noarch 1:13.1.2-1.el7
openstack-nova-compute.noarch 1:13.1.2-1.el7
python-nova.noarch 1:13.1.2-1.el7
python2-novaclient.noarch 1:3.3.2-1.el7
2. Which hypervisor did you use?
Libvirt + KVM
libvirt.x86_64 1.2.17-13.el7_2.5
qemu-kvm.x86_64 10:1.5.3-105.el7_2.7
CentOS Linux release 7.2.1511 (Core)
What's the version of that?
3. Which networking type did you use?
neutron with OpenContrail
Logs & Configs
==============
Instance launch failure exceptions from the compute node:
=========================================================
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [req-c5182216-e2e9-4f8b-a7d4-373f60ac080d e54c8e8fcf4e481d9c108b47fb51834b ac7cde515a7c426986ec1c114fffe02f - - -] [instance: d58c9e56-8610-45f7-be19-2c6649070557] Instance failed to spawn
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] Traceback (most recent call last):
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] yield resources
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] block_device_info=block_device_info)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2779, in spawn
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] block_device_info=block_device_info)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4923, in _create_domain_and_network
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] xml, pause=pause, power_on=power_on)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4849, in _create_domain
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] guest = libvirt_guest.Guest.create(xml, self._host)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 127, in create
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] encodeutils.safe_decode(xml))
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] self.force_reraise()
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] six.reraise(self.type_, self.value, self.tb)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 123, in create
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] domain = host.write_instance_config(xml)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 981, in write_instance_config
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] return self.get_connection().defineXML(xml)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] rv = execute(f, *args, **kwargs)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] six.reraise(c, e, tb)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] rv = meth(*args, **kwargs)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3629, in defineXML
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] libvirtError: invalid argument: could not find capabilities for domaintype=kvm |
Description
===========
Nova doesn't cleanup port when instance launch fails on a hypervisor due to virt issues, instance is automatically rescheduled to a different hypervisor which results in duplicate ports for the instance.
Steps to reproduce
==================
A chronological list of steps which will bring off the
issue you noticed:
* I booted a VM (openstack server create --image cirros --flavor m1.tiny --nic net-id=network-id test-vm)
* then instance was scheduled on hypervisor h1
* then instance launch failed due to virt errors on h1
* then nova did not cleanup the port while cleaning up the failed instance
* then instance was rescheduled on hypervisor h2
* then instance booted successfully
* then I did openstack server list
* then instance showed up with two IPs
* then I did openstack port list
* then there were two ports for the same instance
Expected result
===============
There should only be one port for an instance
Actual result
=============
There were two ports for the instance
Environment
===========
1. Exact version of OpenStack you are running.
openstack-nova-common.noarch 1:13.1.2-1.el7
openstack-nova-compute.noarch 1:13.1.2-1.el7
python-nova.noarch 1:13.1.2-1.el7
python2-novaclient.noarch 1:3.3.2-1.el7
2. Which hypervisor did you use?
Libvirt + KVM
libvirt.x86_64 1.2.17-13.el7_2.5
qemu-kvm.x86_64 10:1.5.3-105.el7_2.7
CentOS Linux release 7.2.1511 (Core)
What's the version of that?
3. Which networking type did you use?
neutron with OpenContrail
Logs & Configs
==============
Instance launch failure exceptions from the compute node:
=========================================================
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [req-c5182216-e2e9-4f8b-a7d4-373f60ac080d e54c8e8fcf4e481d9c108b47fb51834b ac7cde515a7c426986ec1c114fffe02f - - -] [instance: d58c9e56-8610-45f7-be19-2c6649070557] Instance failed to spawn
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] Traceback (most recent call last):
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] yield resources
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] block_device_info=block_device_info)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2779, in spawn
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] block_device_info=block_device_info)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4923, in _create_domain_and_network
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] xml, pause=pause, power_on=power_on)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4849, in _create_domain
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] guest = libvirt_guest.Guest.create(xml, self._host)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 127, in create
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] encodeutils.safe_decode(xml))
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] self.force_reraise()
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] six.reraise(self.type_, self.value, self.tb)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 123, in create
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] domain = host.write_instance_config(xml)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 981, in write_instance_config
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] return self.get_connection().defineXML(xml)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] rv = execute(f, *args, **kwargs)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] six.reraise(c, e, tb)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] rv = meth(*args, **kwargs)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3629, in defineXML
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
2017-05-16 21:25:45.002 8659 ERROR nova.compute.manager [instance: d58c9e56-8610-45f7-be19-2c6649070557] libvirtError: invalid argument: could not find capabilities for domaintype=kvm |
|
