ubuntu libvirtError Unable to open logfile

[Eduardo Gonzalez]

Ubuntu source and binary are currently broken while creating instances:

2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [req-20123bc1-4687-41b3-aeaa-a9dd4cbd78b6 - - - - -] [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] Instance failed to spawn
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] Traceback (most recent call last):
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 2126, in _build_resources
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] yield resources
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 1931, in _build_and_run_instance
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] block_device_info=block_device_info)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2665, in spawn
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] destroy_disks_on_failure=True)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5068, in _create_domain_and_network
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] destroy_disks_on_failure)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] self.force_reraise()
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] six.reraise(self.type_, self.value, self.tb)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5040, in _create_domain_and_network
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] post_xml_callback=post_xml_callback)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4958, in _create_domain
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] guest.launch(pause=pause)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 145, in launch
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] self._encoded_xml, errors='ignore')
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] self.force_reraise()
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] six.reraise(self.type_, self.value, self.tb)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 140, in launch
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] return self._domain.createWithFlags(flags)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] rv = execute(f, *args, **kwargs)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] six.reraise(c, e, tb)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] rv = meth(*args, **kwargs)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] File "/usr/lib/python2.7/site-packages/libvirt.py", line 1065, in createWithFlags
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2017-02-28 15:08:02.729 7 ERROR nova.compute.manager [instance: b51ae956-07b3-49da-9bf3-3994c77c0f46] libvirtError: internal error: process exited while connecting to monitor: 2017-02-28T15:08:02.091842Z qemu-system-x86_64: -chardev pty,id=charserial0,logfile=/var/lib/nova/instances/b51ae956-07b3-49da-9bf3-3994c77c0f46/console.log,logappend=off: Unable to open logfile /var/lib/nova/instances/b51ae956-07b3-49da-9bf3-3994c77c0f46/console.log: Permission denied

[Michał Jastrzębski (inc007)]

From discussion with Jeffrey, it seems to be issue of libvirt 2.5, and that's only version of 2.x shipped with ubuntu. centos has 2.0 and it works well

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/442850

[Jeffrey Zhang (jeffrey4l)]

guess the libvirt 2.5.0 has some issue in "dynamic_ownership".

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/442850
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=b7f8abac296965096d770ee61f8533a162fcb7e4
Submitter: Jenkins
Branch: master

commit b7f8abac296965096d770ee61f8533a162fcb7e4
Author: Michal (inc0) Jastrzebski <email address hidden>
Date: Tue Mar 7 22:57:46 2017 +0000

    Make qemu use nova user

    Libvirt 2.5 (latest version in cloud archive) seems to have changed
    behavior of dynamic_ownership.

    Change-Id: Icf9592c2f82f0f4a35074b9b13a51643a32631e9
    Closes-bug: #1668654

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/443182
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=af9927661dc0a1e514c661da6cf6a7e2b9f90724
Submitter: Jenkins
Branch: stable/ocata

commit af9927661dc0a1e514c661da6cf6a7e2b9f90724
Author: Michal (inc0) Jastrzebski <email address hidden>
Date: Tue Mar 7 22:57:46 2017 +0000

    Make qemu use nova user

    Libvirt 2.5 (latest version in cloud archive) seems to have changed
    behavior of dynamic_ownership.

    Change-Id: Icf9592c2f82f0f4a35074b9b13a51643a32631e9
    Closes-bug: #1668654
    (cherry picked from commit b7f8abac296965096d770ee61f8533a162fcb7e4)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 4.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 5.0.0.0b2 development milestone.

[Giuseppe Paterno' (gpaterno)]

This bug also affect CentOS/RHEL after upgrade to 7.4.
Please specify nova user/group also for centos.
Thanks

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/525891

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/525891
Committed: https://git.openstack.org/cgit/openstack/kolla-ansible/commit/?id=a102cd8efdd76cc9af70827391d6d29445bdd7bd
Submitter: Zuul
Branch: master

commit a102cd8efdd76cc9af70827391d6d29445bdd7bd
Author: Xinliang Liu <email address hidden>
Date: Wed Dec 6 14:03:18 2017 +0800

    Make qemu use nova user on all distros

    This fix libvirtError Unable to open logfile issue.
    Related-Bug: #1668654
    Change-Id: I36720af0c7d3dd7c69d2404843f54c0991aea1bb

[Mike Dorman (mdorman-m)]

Confirmed the above patch fixes this under CentOS 7.4.