Nova fails to launch any instances after the compute host is rebooted

[Paulo Matias]

In Ubuntu 16.04, with stable/newton, Nova fails to launch any instances after the compute host is rebooted. The log displays the following stacktrace:

2016-10-25 15:21:21.426 6385 WARNING nova.virt.libvirt.driver [req-52a13b73-21e3-404c-b8aa-491bc177ecb5 c1bd1be0b75445c497ba46440bd080e0 ea54b5754f70406e848df622513f435b - - -] USB tablet requested for guests by host configuration. In order to accept this request VNC should be enabled or SPICE and SPICE agent disabled on host.
2016-10-25 15:21:21.429 6385 ERROR os_vif [req-52a13b73-21e3-404c-b8aa-491bc177ecb5 c1bd1be0b75445c497ba46440bd080e0 ea54b5754f70406e848df622513f435b - - -] Failed to plug vif VIFBridge(active=False,address=fa:16:3e:f9:a8:6a,bridge_name='brq5d7c8215-a7',has_traffic_filtering=True,id=56efc2e0-b3f0-4962-bff8-e0501b9fc56a,network=Network(5d7c8215-a7a8-42c4-a39f-c4c6210acec0),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap56efc2e0-b3')
2016-10-25 15:21:21.429 6385 ERROR os_vif Traceback (most recent call last):
2016-10-25 15:21:21.429 6385 ERROR os_vif File "/openstack/venvs/nova-14.0.0/lib/python2.7/site-packages/os_vif/__init__.py", line 77, in plug
2016-10-25 15:21:21.429 6385 ERROR os_vif plugin.plug(vif, instance_info)
2016-10-25 15:21:21.429 6385 ERROR os_vif File "/openstack/venvs/nova-14.0.0/lib/python2.7/site-packages/vif_plug_linux_bridge/linux_bridge.py", line 105, in plug
2016-10-25 15:21:21.429 6385 ERROR os_vif linux_net.ensure_bridge(bridge_name, iface)
2016-10-25 15:21:21.429 6385 ERROR os_vif File "/openstack/venvs/nova-14.0.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 264, in inner
2016-10-25 15:21:21.429 6385 ERROR os_vif do_log=False, semaphores=semaphores, delay=delay):
2016-10-25 15:21:21.429 6385 ERROR os_vif File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-10-25 15:21:21.429 6385 ERROR os_vif return self.gen.next()
2016-10-25 15:21:21.429 6385 ERROR os_vif File "/openstack/venvs/nova-14.0.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 216, in lock
2016-10-25 15:21:21.429 6385 ERROR os_vif ext_lock.acquire(delay=delay)
2016-10-25 15:21:21.429 6385 ERROR os_vif File "/openstack/venvs/nova-14.0.0/lib/python2.7/site-packages/fasteners/process_lock.py", line 151, in acquire
2016-10-25 15:21:21.429 6385 ERROR os_vif self._do_open()
2016-10-25 15:21:21.429 6385 ERROR os_vif File "/openstack/venvs/nova-14.0.0/lib/python2.7/site-packages/fasteners/process_lock.py", line 123, in _do_open
2016-10-25 15:21:21.429 6385 ERROR os_vif self.lockfile = open(self.path, 'a')
2016-10-25 15:21:21.429 6385 ERROR os_vif IOError: [Errno 13] Permission denied: '/var/lock/nova/nova-lock_bridge'
2016-10-25 15:21:21.429 6385 ERROR os_vif
2016-10-25 15:21:21.430 6385 ERROR nova.compute.manager [req-52a13b73-21e3-404c-b8aa-491bc177ecb5 c1bd1be0b75445c497ba46440bd080e0 ea54b5754f70406e848df622513f435b - - -] [instance: 2ae563df-8047-4c57-9731-83c778b3fdfe] Instance failed to spawn

If one checks the owner/group of /var/lock/nova, it's root:root, and permissions are 0755. After manually setting the owner to /var/lock/nova, then instances can be launched.

[Paulo Matias (paulo-matias)]

This issue seems not to affect Ubuntu 14.04 according to tests by cloudnull.

After running the playbooks, /var/lock/nova permissions are set correctly: https://github.com/openstack/openstack-ansible-os_nova/blob/0d075aad354b4fd707810ac4de1b3e734a2ebb8a/tasks/nova_pre_install.yml#L72

However, when the host reboots, these permissions are lost.

We need to get the directory defined as the lock_path (https://github.com/openstack/openstack-ansible-os_nova/blob/fa5fbc12f61d1c443780d4ef3347b4e7369744b6/templates/nova.conf.j2#L245) to be created correctly by https://github.com/openstack/openstack-ansible-os_nova/blob/2bb50693c987f7c33fc43dcf38a6dfecde6d9031/templates/nova-systemd-tempfiles.j2

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/391295

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391295
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=1d4c3ad6ece7648d7329c5a6bfd42ed9b53b3022
Submitter: Jenkins
Branch: master

commit 1d4c3ad6ece7648d7329c5a6bfd42ed9b53b3022
Author: Paulo Matias <email address hidden>
Date: Thu Oct 27 17:24:50 2016 -0200

    Fix permissions for the Nova lock directory in systemd

    The directory set as `lock_path` in `nova.conf` must be created with
    the nova system user as its owner, otherwise the VIF plugin would fail
    to acquire a lock when launching an instance.

    Here we set up systemd to create this directory with correct owner and
    permissions, besides moving its path to a `nova_lock_path` variable.

    Apparently, the `/var/lock/{{ item.value.service_name }}` directory,
    which is also created by `nova-systemd-tempfiles.j2`, is not needed at
    all. But as this patch is intended to be backported to a stable release,
    we keep it by now to avoid any unforeseen issues.

    Change-Id: I9886778eddc23f0f71dfdfc87f4f715054946b3d
    Closes-Bug: #1636604

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/391350

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/391350
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_nova/commit/?id=5e7aae8a805947b7f045b08205bd84aff1a0a267
Submitter: Jenkins
Branch: stable/newton

commit 5e7aae8a805947b7f045b08205bd84aff1a0a267
Author: Paulo Matias <email address hidden>
Date: Thu Oct 27 17:24:50 2016 -0200

    Fix permissions for the Nova lock directory in systemd

    The directory set as `lock_path` in `nova.conf` must be created with
    the nova system user as its owner, otherwise the VIF plugin would fail
    to acquire a lock when launching an instance.

    Here we set up systemd to create this directory with correct owner and
    permissions, besides moving its path to a `nova_lock_path` variable.

    Apparently, the `/var/lock/{{ item.value.service_name }}` directory,
    which is also created by `nova-systemd-tempfiles.j2`, is not needed at
    all. But as this patch is intended to be backported to a stable release,
    we keep it by now to avoid any unforeseen issues.

    Change-Id: I9886778eddc23f0f71dfdfc87f4f715054946b3d
    Closes-Bug: #1636604
    (cherry picked from commit 1d4c3ad6ece7648d7329c5a6bfd42ed9b53b3022)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_nova 15.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-ansible-os_nova 14.0.2 release.