OpenStack Compute (nova)

nova destroys wrong instance if trying to create a new instance with the same name

  1. Series pike
  2. Bug #1712460
Bug #1712460 reported by Wonil Choi
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Wonil Choi
Ocata
Fix Released
High
Lee Yarwood
Pike
Fix Released
High
Sean Dague

Bug Description

Description
===========
If trying to create a new VM instance(let's call vm1) with the same instance name
on the same compute host, old existing VM instance(vm1) is destroyed. And openstack
does not intend such situation so the status of old instance(vm1) is 'ACTIVE'

Steps to reproduce
==================
A chronological list of steps which will bring off the
issue you noticed:
1. create one VM.
2. check the VM is created.
3. create another VM on the same host with the same name of the created VM in step 1
  * In order to make same named instance, we can change nova config like below
     instance_name_template = %(hostname)s
4. check the VM exist. openstack has active instance but real VM is not.
:
[controller]# nova list
+--------------------------------------+--------+--------+------------+-------------+----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+----------------------+
| 8595078e-1635-47c3-a027-78436a17f6d4 | testvm | ERROR | - | NOSTATE | demo-net=192.168.0.3 |
| a2239b07-9811-444f-b77e-96756046e8de | testvm | ACTIVE | - | Running | demo-net=192.168.0.7 |
+--------------------------------------+--------+--------+------------+-------------+----------------------+
[compute1]# virsh list
 Id Name State
----------------------------------------------------

Expected result
===============
old VM instance should be running.

Actual result
=============
There is no running VM.

Environment
===========
1. Exact version of OpenStack you are running.
Tested on Ocata

2. Which hypervisor did you use?
Libvirt + KVM

Logs & Configs
==============
# grep instance_name_template /etc/nova/nova.conf
#instance_name_template=instance-%08x
instance_name_template = %(hostname)s

<nova log>
2017-08-21 04:13:11.692 27134 ERROR nova.virt.libvirt.driver [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Failed to start libvirt guest
2017-08-21 04:13:11.693 27134 INFO os_vif [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:ad:3d:09,bridge_name='brq0f056785-2d',has_traffic_filtering=True,id=a74d54bd-04ab-42e1-bbea-2417f487825c,network=Network(0f056785-2d0a-4742-9517-570dbc427e16),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapa74d54bd-04')
2017-08-21 04:13:11.704 27134 INFO nova.virt.libvirt.driver [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Deleting instance files /var/lib/nova/instances/8595078e-1635-47c3-a027-78436a17f6d4_del
2017-08-21 04:13:11.705 27134 INFO nova.virt.libvirt.driver [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Deletion of /var/lib/nova/instances/8595078e-1635-47c3-a027-78436a17f6d4_del complete
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Instance failed to spawn
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Traceback (most recent call last):
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2125, in _build_resources
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] yield resources
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1931, in _build_and_run_instance
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] block_device_info=block_device_info)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2717, in spawn
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] destroy_disks_on_failure=True)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5179, in _create_domain_and_network
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] destroy_disks_on_failure)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] self.force_reraise()
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] six.reraise(self.type_, self.value, self.tb)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5151, in _create_domain_and_network
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] post_xml_callback=post_xml_callback)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5062, in _create_domain
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] guest = libvirt_guest.Guest.create(xml, self._host)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 130, in create
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] encodeutils.safe_decode(xml))
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] self.force_reraise()
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] six.reraise(self.type_, self.value, self.tb)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 126, in create
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] guest = host.write_instance_config(xml)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 835, in write_instance_config
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] domain = self.get_connection().defineXML(xml)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] rv = execute(f, *args, **kwargs)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] six.reraise(c, e, tb)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] rv = meth(*args, **kwargs)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3585, in defineXML
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4] libvirtError: operation failed: domain 'testvm' already exists with uuid a2239b07-9811-444f-b77e-96756046e8de
2017-08-21 04:13:14.105 27134 ERROR nova.compute.manager [instance: 8595078e-1635-47c3-a027-78436a17f6d4]
2017-08-21 04:13:14.108 27134 INFO nova.compute.manager [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Terminating instance
2017-08-21 04:13:14.109 27134 WARNING nova.image.glance [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] No protocol specified in for api_server 'http://128.80.0.6:9292', please update [glance] api_servers with fully qualified url including scheme (http / https)
2017-08-21 04:13:14.329 27134 INFO nova.virt.libvirt.driver [-] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Instance destroyed successfully.
2017-08-21 04:13:14.332 27134 INFO os_vif [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:ad:3d:09,bridge_name='brq0f056785-2d',has_traffic_filtering=True,id=a74d54bd-04ab-42e1-bbea-2417f487825c,network=Network(0f056785-2d0a-4742-9517-570dbc427e16),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tapa74d54bd-04')
2017-08-21 04:13:14.359 27134 INFO nova.virt.libvirt.driver [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Deletion of /var/lib/nova/instances/8595078e-1635-47c3-a027-78436a17f6d4_del complete
2017-08-21 04:13:18.867 27134 WARNING nova.compute.manager [req-eef35b0d-a9a9-44c3-a21e-2cb1fff74224 2d6b9a3227854bfda6ffaca4e1d231b2 843b7144916f44c4adf4cd6290febb51 - - -] [instance: a2239b07-9811-444f-b77e-96756046e8de] Received unexpected event network-vif-unplugged-96e65960-54da-4d41-90b8-23243864d5ba for instance
2017-08-21 04:13:19.503 27134 INFO nova.compute.manager [req-6cf232cd-f038-4d7f-a04e-b9ade01aac8b b0233bb45334440e9bfdd4f67c77e016 abd313825f44461fbfe3cf493d9b2ea4 - - -] [instance: 8595078e-1635-47c3-a027-78436a17f6d4] Took 5.39 seconds to destroy the instance on the hypervisor.

Wonil Choi (wonil22)
Changed in nova:
assignee: nobody → Wonil Choi (wonil22)
Revision history for this message
Wonil Choi (wonil22) wrote :

get_domain() calls conn.lookupByName() in nova/virt/libvirt/driver.py

is there any reason to use lookupByUUIDString() instead of lookupByName() ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/496515

Changed in nova:
status: New → In Progress
Revision history for this message
Tetsuro Nakamura (tetsuro0907) wrote :

I can reproduce the bug with nova(master) with virt_type = qemu.

Changed in nova:
status: In Progress → Confirmed
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/pike)

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/498477

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/498478

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/496515
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ac4705516ada36794c7cc6626875c74c4ef8e547
Submitter: Jenkins
Branch: master

commit ac4705516ada36794c7cc6626875c74c4ef8e547
Author: Wonil Choi <email address hidden>
Date: Wed Aug 23 15:33:02 2017 +0900

    libvirt: Fix getting a wrong guest object

    If trying to create a new VM with the same instance name on the same
    compute host, old existing VM become destroyed. guest object is get by
    instance name so returned object is existing active instance and it is
    destroyed. However this destroying is not intended situation. This
    commit makes get the correct guest object by using UUID.

    Co-Authored-By: Maciej Kucia <email address hidden>
    Change-Id: Ic6f81dc1f8b3610e181914f6d977652cb6d3f6d0
    Closes-Bug: #1712460
    Signed-off-by: Wonil Choi <email address hidden>
    Signed-off-by: Maciej Kucia <email address hidden>

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/pike)

Reviewed: https://review.openstack.org/498477
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=98f0d8162167d1ff6e680adbc622ff7dbadc5a9a
Submitter: Jenkins
Branch: stable/pike

commit 98f0d8162167d1ff6e680adbc622ff7dbadc5a9a
Author: Wonil Choi <email address hidden>
Date: Wed Aug 23 15:33:02 2017 +0900

    libvirt: Fix getting a wrong guest object

    If trying to create a new VM with the same instance name on the same
    compute host, old existing VM become destroyed. guest object is get by
    instance name so returned object is existing active instance and it is
    destroyed. However this destroying is not intended situation. This
    commit makes get the correct guest object by using UUID.

    Co-Authored-By: Maciej Kucia <email address hidden>
    Change-Id: Ic6f81dc1f8b3610e181914f6d977652cb6d3f6d0
    Closes-Bug: #1712460
    Signed-off-by: Wonil Choi <email address hidden>
    Signed-off-by: Maciej Kucia <email address hidden>
    (cherry picked from commit ac4705516ada36794c7cc6626875c74c4ef8e547)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/ocata)

Reviewed: https://review.openstack.org/498478
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5fa3f2ce5c7745c592bfefbd591f31db83a85175
Submitter: Jenkins
Branch: stable/ocata

commit 5fa3f2ce5c7745c592bfefbd591f31db83a85175
Author: Wonil Choi <email address hidden>
Date: Wed Aug 23 15:33:02 2017 +0900

    libvirt: Fix getting a wrong guest object

    If trying to create a new VM with the same instance name on the same
    compute host, old existing VM become destroyed. guest object is get by
    instance name so returned object is existing active instance and it is
    destroyed. However this destroying is not intended situation. This
    commit makes get the correct guest object by using UUID.

    Co-Authored-By: Maciej Kucia <email address hidden>
    Change-Id: Ic6f81dc1f8b3610e181914f6d977652cb6d3f6d0
    Closes-Bug: #1712460
    Signed-off-by: Wonil Choi <email address hidden>
    Signed-off-by: Maciej Kucia <email address hidden>
    (cherry picked from commit ac4705516ada36794c7cc6626875c74c4ef8e547)
    (cherry picked from commit 98f0d8162167d1ff6e680adbc622ff7dbadc5a9a)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 16.0.1

This issue was fixed in the openstack/nova 16.0.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0b1

This issue was fixed in the openstack/nova 17.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.8

This issue was fixed in the openstack/nova 15.0.8 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.