Empty /etc/machine-id file fails instance creation

[Pooja Ghumre]

Description
===========
If the /etc/machine-id file is present but empty for some reason, instance creation fails if the sysinfo_serial config option is set to 'auto' (default). The _get_host_sysinfo_serial_os function in libvirt driver treats empty file the same as missing file and throws NovaException in both cases. This change was made as part of https://bugs.launchpad.net/nova/+bug/1475353.
However, if the sysinfo_serial config option is set to 'auto', the driver only checks for the presence of file when choosing to report 'os' serial. If the file is present but empty, this throws NovaException instead of falling back to reporting 'hardware' serial instead.

Filing this bug to treat empty and missing file as the same in _get_host_sysinfo_serial_auto function in libvirt driver (similar to _get_host_sysinfo_serial_os).

Steps to reproduce
==================
1. Empty the contents of /etc/machine-id file on a KVM host.
2. Set sysinfo_serial option in nova.conf to 'auto' (default).
3. Try to create an instance.

Expected result
===============
Instance creation should be successful.

Actual result
=============
Instance creation failed with error:
NovaException: Unable to get host UUID: /etc/machine-id is empty

Environment
===========
Openstack Liberty
Hypervisor - Libvirt + KVM
Storage - Local disk
Networking - Nova-network

[Pooja Ghumre (pooja-9)]

Posting error stacktrace for reference:

2016-05-16 19:57:09.732 ^[[01;31mERROR nova.compute.manager [req-4e5323f2-8f1b-4471-b166-160043a4bc71 <email address hidden> service^[[01;31m] [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] ^[[01;31mInstance failed to spawn
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] Traceback (most recent call last):
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] File "nova/compute/manager.py", line 2383, in _build_resources
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] yield resources
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] File "nova/compute/manager.py", line 2237, in _build_and_run_instance
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] block_device_info=block_device_info)
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] File "nova/virt/libvirt/driver.py", line 2648, in spawn
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] write_to_disk=True)
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] File "nova/virt/libvirt/driver.py", line 4608, in _get_guest_xml
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] context)
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] File "nova/virt/libvirt/driver.py", line 4457, in _get_guest_config
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] root_device_name)
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] File "nova/virt/libvirt/driver.py", line 4280, in _configure_guest_by_virt_type
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] guest.sysinfo = self._get_guest_config_sysinfo(instance)
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] File "nova/virt/libvirt/driver.py", line 3673, in _get_guest_config_sysinfo
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] sysinfo.system_serial = self._sysinfo_serial_func()
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] File "nova/virt/libvirt/driver.py", line 3662, in _get_host_sysinfo_serial_auto
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] return self._get_host_sysinfo_serial_os()
^[[01;31m2016-05-16 19:57:09.732 TRACE nova.compute.manager [instance: 612120e9-5b3f-44a7-9e7e-2194b87dc459] File "nova/virt/libvirt/driver.py", line 3656, in _get_host_sysinfo_serial_os
^[...

[Sarafraj Singh (sarafraj-singh)]

Please attach logs.

[Markus Zoeller (markus_z) (mzoeller)]

From looking at the code [1] I think the bug report is valid. The flow (in pseudo-code) is:

    if 'auto':
        if '/etc/machine-id' exists:
            if '/etc/machine-id' has content:
                return UUID from file
            else:
                raise NovaException
        else:
            return UUID from host capabilities

The catch of the NovaException is missing to allow a fallback to the hardware host capabilites.

References:
[1] https://github.com/openstack/nova/blob/660ecaee66ccab895b282c2ed45c95c809ad6833/nova/virt/libvirt/driver.py#L3589-L3593

[Markus Zoeller (markus_z) (mzoeller)]

@Pooja Ghumre: You're set as assignee, which usually means people expect you to push a code change for this bug in the very near future. Was this your intention when you set yourself as assignee? If that's not the case, just remove yourself as assignee. Other folks will then see that the can pick this up.

[Pooja Ghumre (pooja-9)]

Hi Markus, I intend to work on this bug fix and submit it for review in a couple of days.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/343943

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.opendev.org/343943
Reason: Wow this is old and stale. If anyone cares to continue working on this we can restore and rebase.