can not create VM instance due to permission problem

[Hui Kang]

Tenance VM failed to be created using ubuntu source build type. The error message from nova_compute container is

2016-01-12 20:22:59.766 1 DEBUG nova.compute.manager [req-6a8701b3-10f1-4ebe-a4e7-4c42f541abe9 70c5cff5b3704338a3e398fd0d89f528 0d0f81a8af2e4e0f94eb3df96fc730b4 - - -] [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] Start spawning the instance on the hypervisor. _build_and_run_instance /var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py:2024
2016-01-12 20:22:59.767 1 DEBUG nova.block_device [req-6a8701b3-10f1-4ebe-a4e7-4c42f541abe9 70c5cff5b3704338a3e398fd0d89f528 0d0f81a8af2e4e0f94eb3df96fc730b4 - - -] block_device_list [] volume_in_mapping /var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/block_device.py:584
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [req-6a8701b3-10f1-4ebe-a4e7-4c42f541abe9 70c5cff5b3704338a3e398fd0d89f528 0d0f81a8af2e4e0f94eb3df96fc730b4 - - -] [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] Instance failed to spawn
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] Traceback (most recent call last):
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2182, in _build_resources
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] yield resources
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2029, in _build_and_run_instance
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] block_device_info=block_device_info)
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2510, in spawn
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] admin_pass=admin_password)
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2873, in _create_image
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] fileutils.ensure_tree(libvirt_utils.get_instance_path(instance))
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/fileutils.py", line 42, in ensure_tree
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] os.makedirs(path, mode)
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] File "/var/lib/kolla/venv/lib/python2.7/os.py", line 157, in makedirs
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] mkdir(name, mode)
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/15a7820d-f01a-4da3-a94f-180ead74f752'
2016-01-12 20:22:59.768 1 ERROR nova.compute.manager [instance: 15a7820d-f01a-4da3-a94f-180ead74f752]

[Artur Zarzycki (azarzycki)]

As I see on my build /var/lib/nova has good rights (owner nova) but subdirectory in it /var/lib/nova/instances is owned by root.
Does this bug affects you on centos or ubuntu installation or both?

[Artur Zarzycki (azarzycki)]

I confirm that error exist on in both distributions.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/267015

[OpenStack Infra (hudson-openstack)]

Change abandoned by Artur Zarzycki (<email address hidden>) on branch: master
Review: https://review.openstack.org/267015

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/266575
Committed: https://git.openstack.org/cgit/openstack/kolla/commit/?id=ef971bff5135b88e3451eb451418327e672f6d8b
Submitter: Jenkins
Branch: master

commit ef971bff5135b88e3451eb451418327e672f6d8b
Author: Artur Zarzycki <email address hidden>
Date: Wed Jan 13 17:28:53 2016 +0100

    Fix permissions to /var/lib/nova/

    Due to changes with the drop-root work, we lost the ability to write
    to /var/lib/nova/*. This fixes those permissions and ensures cross
    container talk works properly between nova_libvirt and nova_compute

    Additionally, this fixes another issue introduced which saw that
    nova-compute could not run sudo commands as it did not have a proper
    sudoers entry

    Testing from previous deploys means you need a fresh environment. You
    have to remove all of the named volumes that kolla created in docker.
    Check these with `docker volume ls`

    Signed-off-by: Hui Kang <email address hidden>
    Signed-off-by: Artur Zarzycki <email address hidden>
    Co-Authored-By: Sam Yaple <email address hidden>
    Co-Authored-By: Hui Kang <email address hidden>
    Closes-Bug: #1533350
    Change-Id: I7f864c448a2414e0b5d89f48337be411b891df35