periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 on master/victoria are failing on tempest test while spawning instance with Error - Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file."

[Sandeep Yadav]

Description:-
==========

periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master is failing on tempest test while spawning instance:-
~~~
ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-11-08T19:22:02.094817Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.
~~~

Tempest failure:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master/93d0738/logs/undercloud/var/log/tempest/tempest_run.log.txt.gz

~~~
tempest.exceptions.BuildErrorException: Server 2f420465-c919-4714-a42a-55416654a297 failed to build and is in ERROR status
    Details: {'code': 500, 'created': '2020-11-08T19:17:46Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 2f420465-c919-4714-a42a-55416654a297.'}
~~~

Error snippet from nova:-

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-master/93d0738/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz
~~~
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [req-3da86b99-ecc9-4d5b-836f-9aeca0cea983 bf541eaf96594646b5336c58cb8bb856 0cf010a9cd1b475a9ab5f271db22477c - default default] [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] Failed to build and run instance: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-11-08T19:22:02.094817Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] Traceback (most recent call last):
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 2393, in _build_and_run_instance
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] accel_info=accel_info)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 3679, in spawn
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] cleanup_instance_disks=created_disks)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6661, in _create_guest_with_network
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] cleanup_instance_disks=cleanup_instance_disks)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] self.force_reraise()
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] six.reraise(self.type_, self.value, self.tb)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] raise value
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6630, in _create_guest_with_network
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] post_xml_callback=post_xml_callback)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 6560, in _create_guest
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] guest.launch(pause=pause)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 159, in launch
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] self._encoded_xml, errors='ignore')
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] self.force_reraise()
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] six.reraise(self.type_, self.value, self.tb)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] raise value
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py", line 155, in launch
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] return self._domain.createWithFlags(flags)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 190, in doit
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 148, in proxy_call
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] rv = execute(f, *args, **kwargs)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 129, in execute
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] six.reraise(c, e, tb)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] raise value
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib/python3.6/site-packages/eventlet/tpool.py", line 83, in tworker
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] rv = meth(*args, **kwargs)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] File "/usr/lib64/python3.6/site-packages/libvirt.py", line 1265, in createWithFlags
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2020-11-08 19:22:02.615 8 ERROR nova.compute.manager [instance: d20f9855-727e-4daa-927a-fbd7bd76ab7a] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-11-08T19:22:02.094817Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.
~~~

[Sandeep Yadav (sandeepyadav93)]

Hello,

Featureset039 is also affected with the same error.

https://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039-master/26f4e84/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.txt.gz
~~~
2020-11-12 04:02:36.528 7 ERROR nova.virt.libvirt.guest libvirt.libvirtError: internal error: process exited while connecting to monitor: 2020-11-12T04:02:36.179478Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.
2020-11-12 04:02:36.528 7 ERROR nova.virt.libvirt.guest
~~~

periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa and featureset039 are also failing on victoria branch.

https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-victoria/1bf7f31/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz

~~~
2020-11-11 11:45:05.748 7 WARNING oslo_privsep.priv_context [req-3bef255f-d9ca-45ae-8c57-83268bf4c3a4 22751d4280194fa49f9b8c9c14bb0738 e4a59df6fbd64223bfeca8774d5244f5 - default default] privsep daemon already running: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-11-11T11:45:04.314113Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.
~~~

[Martin Schuppert (mschuppert)]

from e.g. https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-victoria/2858887/logs/undercloud/var/log/extra/podman/containers/nova_libvirt/podman_info.log.txt.gz the bind mounts are configured for the libvirt container:

        "HostConfig": {
            "Binds": [
                "/var/lib/config-data/puppet-generated/nova_libvirt:/var/lib/kolla/config_files/src:ro,rprivate,rbind",
                "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro,rprivate,rbind",
                "/var/log/libvirt/qemu:/var/log/libvirt/qemu:ro,rprivate,rbind",
                "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro,rprivate,rbind",
                "/etc/pki/qemu/server-cert.pem:/etc/pki/qemu/server-cert.pem:ro,rprivate,rbind",
                "/var/log/containers/libvirt:/var/log/libvirt:rw,rprivate,rbind",
                "/var/lib/libvirt:/var/lib/libvirt:shared,rw,rbind",
                "/etc/pki/libvirt-vnc/server-key.pem:/etc/pki/libvirt-vnc/server-key.pem:ro,rprivate,rbind",
                "/sys/fs/cgroup:/sys/fs/cgroup:rw,rprivate,noexec,nosuid,nodev,rbind",
                "/etc/pki/libvirt-vnc/server-cert.pem:/etc/pki/libvirt-vnc/server-cert.pem:ro,rprivate,rbind",
                "/var/lib/vhost_sockets:/var/lib/vhost_sockets:rw,rprivate,rbind",
                "/etc/ipa/ca.crt:/etc/pki/CA/cacert.pem:ro,rprivate,rbind",
                "/etc/pki/libvirt-nbd:/etc/pki/libvirt-nbd:ro,rprivate,rbind",
                "/etc/hosts:/etc/hosts:ro,rprivate,rbind",
                "/run:/run:rw,rprivate,nosuid,nodev,rbind",
                "/var/cache/libvirt:/var/cache/libvirt:shared,rw,rbind",
                "/var/lib/nova:/var/lib/nova:shared,rw,rbind",
                "/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro,rprivate,rbind",
                "/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro,rprivate,rbind",
                "/etc/ceph:/var/lib/kolla/config_files/src-ceph:ro,rprivate,rbind",
                "/etc/selinux/config:/etc/selinux/config:ro,rprivate,rbind",
                "/lib/modules:/lib/modules:ro,rprivate,rbind",
                "/etc/pki/qemu/ca-cert.pem:/etc/pki/qemu/ca-cert.pem:ro,rprivate,rbind",
                "/sys/fs/selinux:/sys/fs/selinux:rw,rprivate,rbind",
                "/etc/localtime:/etc/localtime:ro,rprivate,rbind",
                "/var/lib/container-config-scripts/nova_libvirt_launcher.sh:/nova_libvirt_launcher.sh:ro,rprivate,rbind",
                "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro,rprivate,rbind",
                "/dev/log:/dev/log:rw,rprivate,nosuid,rbind",
                "/run/libvirt:/run/libvirt:shared,rw,nosuid,nodev,rbind",
                "/dev:/dev:rw,rprivate,nosuid,rbind",
                "/etc/pki/CA/certs/vnc.crt:/etc/pki/libvirt-vnc/ca-cert.pem:ro,rprivate,rbind",
                "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro,rprivate,rbind",
                "/etc/pki/libvirt:/etc/pki/libvirt:ro,rprivate,rbind",
                "/etc/libvirt:/etc/libvirt:rw,rprivate,rbind",
                "/etc/pki/tls/certs/ca...

[Martin Schuppert (mschuppert)]

From reproducer env:

[zuul@node-0000586121 ~]$ ll /etc/pki/libvirt-vnc/
total 16
lrwxrwxrwx. 1 root root 31 Nov 12 09:09 ca-cert.pem -> /etc/pki/CA/certs/vnc-proxy.crt
-rw-r--r--. 1 root root 1911 Nov 12 09:09 client-cert.pem
-rw-r-----. 1 root qemu 1708 Nov 12 09:09 client-key.pem
-rw-r--r--. 1 root root 1911 Nov 12 09:09 server-cert.pem
-rw-r-----. 1 root qemu 1704 Nov 12 09:09 server-key.pem

[zuul@node-0000586121 ~]$ sudo podman exec -it -u root nova_libvirt sh
sh-4.4# ls -la /etc/pki
total 88
drwxr-xr-x. 1 root root 4096 Nov 12 09:21 .
drwxr-xr-x. 1 root root 4096 Nov 12 09:21 ..
drwxr-xr-x. 1 root root 4096 Nov 12 09:21 CA
drwxr-xr-x. 4 root root 4096 Sep 1 19:39 ca-trust
drwxr-xr-x. 2 root root 4096 Aug 17 20:21 consumer
drwxr-xr-x. 2 root root 4096 Aug 17 20:21 entitlement
lrwxrwxrwx. 1 root root 32 Sep 1 19:39 entitlement-host -> /run/secrets/etc-pki-entitlement
drwxr-xr-x. 2 root root 4096 Sep 1 19:39 java
drwxr-xr-x. 3 root root 4096 Nov 12 09:09 libvirt
drwxr-xr-x. 2 root root 4096 Nov 12 09:09 libvirt-nbd
drwxr-xr-x. 2 root root 4096 Nov 12 09:21 libvirt-vnc
drwxr-xr-x. 2 root root 4096 Nov 12 00:49 nssdb
drwxr-xr-x. 2 root root 4096 Nov 12 00:31 product
drwxr-xr-x. 2 root root 4096 Sep 1 19:39 product-default
drwxr-xr-x. 2 root root 4096 Nov 12 09:21 qemu
drwxr-xr-x. 2 root root 4096 Sep 1 19:39 rpm-gpg
drwxr-xr-x. 3 root root 4096 Sep 1 19:39 swid
drwxr-xr-x. 5 root root 4096 Sep 1 19:39 tls

sh-4.4# ls -la /etc/pki/libvirt-vnc/
total 28
drwxr-xr-x. 2 root root 4096 Nov 12 09:21 .
drwxr-xr-x. 1 root root 4096 Nov 12 09:21 ..
-rw-r--r--. 1 root root 1623 Nov 12 09:09 ca-cert.pem
-rw-r--r--. 1 root root 1911 Nov 12 09:09 server-cert.pem
-rw-r-----. 1 root 107 1704 Nov 12 09:09 server-key.pem

The issue is that the qemu user id is different from the one on the host:

sh-4.4# grep qemu /etc/passwd
qemu:x:42427:42427::/home/qemu:/usr/sbin/nologin

This is different then in previous releases.

From a downstream osp16 (train) release:

[root@compute-0 ~]# podman exec -it -u root nova_libvirt sh
()[root@compute-0 /]$ grep qemu /etc/passwd
qemu:x:107:107::/home/qemu:/usr/sbin/nologin

[Martin Schuppert (mschuppert)]

qemu user id in the new container image build process is different from what we had before:
https://github.com/openstack/tripleo-common/blob/master/container-images/kolla/base/uid_gid_manage.sh#L66

https://github.com/openstack/tripleo-common/commit/fcdca87024d99e3eac61bdb227f179b286ca6e04

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/762497

[Marios Andreou (marios-b)]

also impacting victoria - seen in periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-victoria from last night's run:

        * https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-victoria/98642ec/logs/undercloud/var/log/tempest/stestr_results.html.gz
        * tempest.exceptions.BuildErrorException: Server 133a0a31-40ee-4426-816f-55169e45ffa3 failed to build and is in ERROR status
Details: {'code': 500, 'created': '2020-11-12T23:50:15Z', 'message': 'Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 133a0a31-40ee-4426-816f-55169e45ffa3.'}
        * https://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-standalone-on-multinode-ipa-victoria/98642ec/logs/undercloud/var/log/containers/nova/nova-compute.log.txt.gz
        * : libvirt.libvirtError: internal error: process exited while connecting to monitor: 2020-11-12T23:49:32.637750Z qemu-kvm: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/libvirt-vnc,endpoint=server,verify-peer=yes: Cannot load certificate '/etc/pki/libvirt-vnc/server-cert.pem' & key '/etc/pki/libvirt-vnc/server-key.pem': Error while reading file.

so this is also victoria promotion blocker

[Martin Schuppert (mschuppert)]

Yes I am aware of this. We need it all the way back to train where the container image
change happened. I have stopped backporting https://review.opendev.org/#/c/762306/
till then.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/762497
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=222c67e128620cfd4cc0d65ddd15d73991d486d6
Submitter: Zuul
Branch: master

commit 222c67e128620cfd4cc0d65ddd15d73991d486d6
Author: Martin Schuppert <email address hidden>
Date: Thu Nov 12 11:44:55 2020 +0100

    Change qemu user id to match previous releases

    The qemu user on the host gets created using uid/gid 107. Certificates
    on the host, but also the vhost-user sockets created by ovs use this
    uid/gid. With the move to TCIB images the default kolla id were
    reverted and the previous overwrite dropped. This make e.g. the qemu
    processes to fail to use the libvirt-vnc bind mounted certificates.
    This change brings back the previous overwrite of the qemu user
    uid/gid.

    Closes-Bug: #1903508
    Related-Bug: #1900986

    Change-Id: I54b9d9f341b521b415a6dccc6c78ae7a77821f6f

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/763185

[Marios Andreou (marios-b)]

because there was a merge and the bug went automatically to fix-released adding a comment...

we aren't done yet because we need this cherry-picked back to train per https://bugs.launchpad.net/tripleo/+bug/1903508/comments/7

it is also impacting victoria for example and should be fixed with the cherrypick at https://review.opendev.org/763185 (that is currently blocked on https://review.opendev.org/#/c/763005 fighting its way through the gate)

[wes hayutin (weshayutin)]

waiting on train https://review.opendev.org/c/openstack/tripleo-common/+/763721

[Martin Schuppert (mschuppert)]

ussuri merged https://review.opendev.org/c/openstack/tripleo-common/+/763720

[Marios Andreou (marios-b)]

still waiting on the train cherrypick https://review.opendev.org/c/openstack/tripleo-common/+/763721

otherwise this is done.

[Marios Andreou (marios-b)]

moving back to in progress until the train cherrypick merges https://review.opendev.org/c/openstack/tripleo-common/+/763721 still fighting the gate battle

[Marios Andreou (marios-b)]

https://review.opendev.org/c/openstack/tripleo-common/+/763721 is blocked on https://review.opendev.org/c/openstack/tripleo-common/+/766188 for https://bugs.launchpad.net/tripleo/+bug/1907236

[wes hayutin (weshayutin)]

closing, all patches merged.. will reopen if needed.
Thanks all :)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 13.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 14.0.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 12.4.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 11.5.0 release.