Description:-
==========
periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master is failing on tempest test while spawning instance:-
~~~
ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-11-08T19:22:02.094817Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.
~~~
Tempest failure:-
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master/93d0738/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz
~~~
tempest.exceptions.BuildErrorException: Server 2f420465-c919-4714-a42a-55416654a297 failed to build and is in ERROR status
Details: {'code': 500, 'created': '2020-11-08T19:17:46Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2f420465-c919-4714-a42a-55416654a297.'}
~~~
Error snippet from nova:-
https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master/93d0738/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz
~~~
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [req-3da86b99-ecc9-4d5b-836f-9aeca0cea983 bf541eaf96594646b5336c58cb8bb856 0cf010a9cd1b475a9ab5f271db22477c - default default] [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] Failed to build and run instance: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-11-08T19:22:02.094817Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] Traceback (most recent call last):
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2393, in _build_and_run_instance
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] accel_info=accel_info)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3679, in spawn
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] cleanup_instance_disks=created_disks)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6661, in _create_guest_with_network
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] cleanup_instance_disks=cleanup_instance_disks)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] self.force_reraise()
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] six.reraise(self.type_, self.value, self.tb)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] raise value
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6630, in _create_guest_with_network
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] post_xml_callback=post_xml_callback)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6560, in _create_guest
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] guest.launch(pause=pause)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] self._encoded_xml, errors='ignore')
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] self.force_reraise()
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] six.reraise(self.type_, self.value, self.tb)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] raise value
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 155, in launch
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] return self._domain.createWithFlags(flags)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] rv = execute(f, *args, **kwargs)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] six.reraise(c, e, tb)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] raise value
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] rv = meth(*args, **kwargs)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1265, in createWithFlags
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-11-08T19:22:02.094817Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.
~~~
Hello,
Featureset039 is also affected with the same error.
https:/ /logserver. rdoproject. org/openstack- periodic- integration- main/opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 8-ovb-3ctlr_ 1comp_1supp- featureset039- master/ 26f4e84/ logs/overcloud- novacompute- 0/var/log/ containers/ nova/nova- compute. log.txt. gz libvirt. guest libvirt. libvirtError: internal error: process exited while connecting to monitor: 2020-11- 12T04:02: 36.179478Z qemu-kvm: -object tls-creds- x509,id= vnc-tls- creds0, dir=/etc/ pki/libvirt- vnc,endpoint= server, verify- peer=yes: Cannot load certificate '/etc/pki/ libvirt- vnc/server- cert.pem' & key '/etc/pki/ libvirt- vnc/server- key.pem' : Error while reading file. libvirt. guest
~~~
2020-11-12 04:02:36.528 7 ERROR nova.virt.
2020-11-12 04:02:36.528 7 ERROR nova.virt.
~~~
periodic- tripleo- ci-centos- 8-standalone- on-multinode- ipa and featureset039 are also failing on victoria branch.
https:/ /logserver. rdoproject. org/openstack- periodic- integration- stable1/ opendev. org/openstack/ tripleo- ci/master/ periodic- tripleo- ci-centos- 8-standalone- on-multinode- ipa-victoria/ 1bf7f31/ logs/undercloud /var/log/ containers/ nova/nova- compute. log.txt. gz
~~~ priv_context [req-3bef255f- d9ca-45ae- 8c57-83268bf4c3 a4 22751d4280194fa 49f9b8c9c14bb07 38 e4a59df6fbd6422 3bfeca8774d5244 f5 - default default] privsep daemon already running: libvirt. libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-11- 11T11:45: 04.314113Z qemu-kvm: -object tls-creds- x509,id= vnc-tls- creds0, dir=/etc/ pki/libvirt- vnc,endpoint= server, verify- peer=yes: Cannot load certificate '/etc/pki/ libvirt- vnc/server- cert.pem' & key '/etc/pki/ libvirt- vnc/server- key.pem' : Error while reading file.
2020-11-11 11:45:05.748 7 WARNING oslo_privsep.
~~~