Instances with non-ASCII encoded names are not resumed when the host reboots

[Fabrizio Soppelsa]

Seen on 5.1 and 6.0-42-2014-10-23_12-48-42 with libvirtd.

When in nova.conf this option is enabled:

resume_guests_state_on_host_boot=True

VMs are not resumed correctly after a compute node reboot, if their name contains non-ASCII characters:

root@node-1:~# nova list
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------+
| 2a3735c1-117a-4e18-97fa-e8ff086e7d59 | cirros1 | ACTIVE | - | Running | net04=192.168.111.3; net04_ext=172.16.0.142 |
| 0d2080c8-2bbb-4343-9fa8-e1ad1936e27c | cirros2 | ACTIVE | - | Running | net04=192.168.111.2; net04_ext=172.16.0.141 |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------+
root@node-2:~# virsh list
 Id Name State
----------------------------------------------------
 2 instance-00000001 running
 3 instance-00000002 running

root@node-2:~# reboot

root@node-2:~# virsh list
 Id Name State
----------------------------------------------------
 2 instance-00000001 running
 3 instance-00000002 running

When the instance name contains non-ASCII characters, the instance is not resumed:

root@node-1:~# nova rename cirros2 cirròs
root@node-1:~# nova list
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------+
| 2a3735c1-117a-4e18-97fa-e8ff086e7d59 | cirros1 | ACTIVE | - | Running | net04=192.168.111.3; net04_ext=172.16.0.142 |
| 0d2080c8-2bbb-4343-9fa8-e1ad1936e27c | cirròs | ACTIVE | - | Shutdown | net04=192.168.111.2; net04_ext=172.16.0.141 |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------+

root@node-2:~# reboot

root@node-1:~# nova list
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------+
| 2a3735c1-117a-4e18-97fa-e8ff086e7d59 | cirros1 | ACTIVE | - | Running | net04=192.168.111.3; net04_ext=172.16.0.142 |
| 0d2080c8-2bbb-4343-9fa8-e1ad1936e27c | cirròs | ERROR | - | Shutdown | net04=192.168.111.2; net04_ext=172.16.0.141 |
+--------------------------------------+---------+--------+------------+-------------+---------------------------------------------+

root@node-2:~# virsh list
 Id Name State
----------------------------------------------------
 2 instance-00000002 running

On 5.1, all instances on the host are not resumed.
On 6.0-42-2014-10-23_12-48-42, only the one with weird chars is not resumed.

Extract of nova-compute.log on 5.1:

2014-10-21 16:25:09.354 2593 AUDIT nova.service [-] Starting compute node (version 2014.1.1)
2014-10-21 16:25:10.529 2593 INFO nova.compute.manager [-] [instance: c0119920-e839-44ca-a20e-38d7c515c4e9] Rebooting instance after nova-compute restart.
2014-10-21 16:25:10.560 2593 INFO nova.virt.libvirt.driver [-] [instance: c0119920-e839-44ca-a20e-38d7c515c4e9] Instance destroyed successfully.
2014-10-21 16:25:10.631 2593 WARNING nova.compute.utils [-] [instance: c0119920-e839-44ca-a20e-38d7c515c4e9] Can't access image : <type 'NoneType'> can't be encoded
2014-10-21 16:25:11.396 2593 INFO nova.compute.manager [-] Lifecycle event 0 on VM c0119920-e839-44ca-a20e-38d7c515c4e9
2014-10-21 16:25:11.403 2593 INFO nova.virt.libvirt.driver [-] [instance: c0119920-e839-44ca-a20e-38d7c515c4e9] Instance rebooted successfully.

[Stanislav Makar (smakar)]

Confirmed on 6.0
This is not only the problem with instances, which names are in non-ASCII, after reboot
I can not even start the instances with names in non-ASCII

| fault | {"message": "No valid host was found. ", "code": 500, "details": " File \"/usr/lib/python2.7/dist-packages/nova/conductor/manager.py\", line 614, in build_instances |
| | request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py\", line 49, in select_destinations |
| | context, request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py\", line 35, in __run_method |
| | return getattr(self.instance, __name)(*args, **kwargs) |
| | File \"/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py\", line 34, in select_destinations |
| | context, request_spec, filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py\", line 107, in select_destinations |
| | request_spec=request_spec, filter_properties=filter_properties) |
| | File \"/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py\", line 152, in call |
| | retry=self.retry) |
| | File \"/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py\", line 90, in _send |
| | timeout=timeout, retry=retry) |
| | File \"/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/am...

[Stanislav Makar (smakar)]

I have found an error:

2014-10-24 15:18:09.530 3338 ERROR nova.compute.manager [req-55bb5359-3794-4676-845d-096562208c0c None] [instance: af34c806-96dd-4d95-9452-8c964fce4aee] Instance failed to spawn
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] Traceback (most recent call last):
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2231, in _build_resources
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] yield resources
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2101, in _build_and_run_instance
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] block_device_info=block_device_info)
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2617, in spawn
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] write_to_disk=True)
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4149, in _get_guest_xml
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] xml = conf.to_xml()
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 78, in to_xml
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] root = self.format_dom()
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 1680, in format_dom
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] self._format_basic_props(root)
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 1619, in _format_basic_props
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] metadata.append(m.format_dom())
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/config.py", line 1909, in format_dom
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] meta.append(self._text_node("name", self.name))
2014-10-24 15:18:09.530 3338 TRACE nova.compute.manager [instance: af34c806-96dd-4d95-9452-8c964fce4aee] File "/usr/lib/python2.7/dist...

[Roman Podoliaka (rpodolyaka)]

Fixed by merging the upstream in https://review.fuel-infra.org/#/c/602/

[Alexander Gubanov (ogubanov)]

Bug still reproducing on mos 6.0 (build_number: '110')

detail info http://paste.openstack.org/show/134062/

[Roman Podoliaka (rpodolyaka)]

Alexander, this is a different failure. I filed a separate bug for it - https://bugs.launchpad.net/mos/+bug/1396156

[Alexander Gubanov (ogubanov)]

I have verified it on mos 6.1 (build 126) - fixed!
Proof:

root@node-1:~# nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| c08cf9e9-8eca-4b02-9b3e-ec23b3373817 | тест2 | ACTIVE | - | Running | net04=192.168.111.6 |
+--------------------------------------+-------+--------+------------+-------------+---------------------+

root@node-1:~# nova show c08cf9e9-8eca-4b02-9b3e-ec23b3373817 | grep hypervisor_hostname
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-5.test.domain.local

Nowadays, option resume_guests_state_on_host_boot by default set in True

root@node-5:~# grep 'resume_guests_state_on_host_boot' /etc/nova/nova.conf
resume_guests_state_on_host_boot=True

root@node-5:~# reboot
root@node-5:~#
Broadcast message from root@node-5
        (/dev/pts/1) at 11:39 ...

The system is going down for reboot NOW!
Connection to 10.108.20.3 closed by remote host.
Connection to 10.108.20.3 closed.

root@node-1:~# nova list
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+---------------------+
| c08cf9e9-8eca-4b02-9b3e-ec23b3373817 | тест2 | ACTIVE | - | Running | net04=192.168.111.6 |
+--------------------------------------+-------+--------+------------+-------------+---------------------+