Deployment with TLS fails, 'Error: Could not find group qemu'

[Rajesh Tailor]

Description of problem:
Deployment with TLS fails.
Ansible failed on all controllers

tail -n 10 ansible.log
2020-01-26 08:53:13,746 p=756 u=mistral | NO MORE HOSTS LEFT *************************************************************
2020-01-26 08:53:13,748 p=756 u=mistral | PLAY RECAP *********************************************************************
2020-01-26 08:53:13,748 p=756 u=mistral | compute-0 : ok=170 changed=95 unreachable=0 failed=0 skipped=90 rescued=0 ignored=1
2020-01-26 08:53:13,748 p=756 u=mistral | compute-1 : ok=165 changed=93 unreachable=0 failed=0 skipped=90 rescued=0 ignored=1
2020-01-26 08:53:13,748 p=756 u=mistral | controller-0 : ok=223 changed=137 unreachable=0 failed=1 skipped=84 rescued=0 ignored=0
2020-01-26 08:53:13,748 p=756 u=mistral | controller-1 : ok=208 changed=137 unreachable=0 failed=1 skipped=84 rescued=0 ignored=0
2020-01-26 08:53:13,748 p=756 u=mistral | controller-2 : ok=208 changed=137 unreachable=0 failed=1 skipped=84 rescued=0 ignored=0
2020-01-26 08:53:13,749 p=756 u=mistral | undercloud : ok=11 changed=7 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
2020-01-26 08:53:13,819 p=756 u=mistral | Sunday 26 January 2020 08:53:13 +0000 (0:01:42.458) 0:10:10.548 ********
2020-01-26 08:53:13,819 p=756 u=mistral | ===============================================================================

cat ansible-errors.json | sed 's/Jan/\nJan/g' | grep -i error
Jan 26 08:51:36 puppet-user: Debug: Facter: Error: unable to get cib\n<13>
Jan 26 08:51:45 puppet-user: Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/unless: Error: cluster is not currently running on this node\n<13>
Jan 26 08:52:41 puppet-user: Error: Could not find group qemu\n<13>
Jan 26 08:52:41 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-client-cert]/File[/etc/pki/libvirt-vnc/client-key.pem]/group: change from 'root' to 'qemu' failed: Could not find group qemu\n<13>
Jan 26 08:51:36 puppet-user: Debug: Facter: Error: unable to get cib", "<13>
Jan 26 08:51:45 puppet-user: Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/unless: Error: cluster is not currently running on this node", "<13>
Jan 26 08:52:41 puppet-user: Error: Could not find group qemu", "<13>
Jan 26 08:52:41 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-client-cert]/File[/etc/pki/libvirt-vnc/client-key.pem]/group: change from 'root' to 'qemu' failed: Could not find group qemu", "<13>
Jan 26 08:51:36 puppet-user: Debug: Facter: Error: unable to get cib\n<13>
Jan 26 08:51:38 puppet-user: error: Could not connect to cluster (is it running?)\n<13>
Jan 26 08:52:42 puppet-user: Error: Could not find group qemu\n<13>
Jan 26 08:52:42 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-client-cert]/File[/etc/pki/libvirt-vnc/client-key.pem]/group: change from 'root' to 'qemu' failed: Could not find group qemu\n<13>
Jan 26 08:51:36 puppet-user: Debug: Facter: Error: unable to get cib", "<13>
Jan 26 08:51:38 puppet-user: error: Could not connect to cluster (is it running?)", "<13>
Jan 26 08:52:42 puppet-user: Error: Could not find group qemu", "<13>
Jan 26 08:52:42 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-client-cert]/File[/etc/pki/libvirt-vnc/client-key.pem]/group: change from 'root' to 'qemu' failed: Could not find group qemu", "<13>
Jan 26 08:51:36 puppet-user: Debug: Facter: Error: unable to get cib\n<13>
Jan 26 08:51:45 puppet-user: Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/unless: Error: cluster is not currently running on this node\n<13>
Jan 26 08:52:52 puppet-user: Error: Could not find group qemu\n<13>
Jan 26 08:52:52 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-client-cert]/File[/etc/pki/libvirt-vnc/client-key.pem]/group: change from 'root' to 'qemu' failed: Could not find group qemu\n<13>
Jan 26 08:51:36 puppet-user: Debug: Facter: Error: unable to get cib", "<13>
Jan 26 08:51:45 puppet-user: Debug: /Stage[main]/Pacemaker::Corosync/Exec[wait-for-settle]/unless: Error: cluster is not currently running on this node", "<13>
Jan 26 08:52:52 puppet-user: Error: Could not find group qemu", "<13>
Jan 26 08:52:52 puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Libvirt_vnc[libvirt-vnc-client-cert]/File[/etc/pki/libvirt-vnc/client-key.pem]/group: change from 'root' to 'qemu' failed: Could not find group qemu", "<13>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/704303

[wes hayutin (weshayutin)]

This started to fail on Jan 23/24

https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-train

This looks like it needs a revert
https://review.opendev.org/#/c/702837/

revert is here:
https://review.opendev.org/#/c/704349/

[wes hayutin (weshayutin)]

additional train revert here: https://review.opendev.org/#/c/704350/

[chandan kumar (chkumar246)]

From recent run http://logs.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-7-ovb-3ctlr_1comp_1supp-featureset039-master/69c7f4f/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz It is seen in master also.

Fs039 train job got hit with post_failure.

https://review.opendev.org/#/c/682255/ needs to be reverted in master run.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/704748

[chandan kumar (chkumar246)]

https://review.opendev.org/#/c/704748/ added the master revert and testing here https://review.rdoproject.org/r/#/c/24682/

[Rajesh Tailor (ratailor)]

Instead of reverting in master, we should fix it with https://review.opendev.org/#/c/704303/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chandan Kumar (raukadah) (<email address hidden>) on branch: master
Review: https://review.opendev.org/704748
Reason: Thanks rajesh testing with above patch

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/704303
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b8c6154e7afda5525be00189afde15dac649a7b2
Submitter: Zuul
Branch: master

commit b8c6154e7afda5525be00189afde15dac649a7b2
Author: Rajesh Tailor <email address hidden>
Date: Mon Jan 27 16:20:26 2020 +0530

    Create qemu user/group on controller

    Deployment is failing with error [1] because the owner/group
    of the TLS generated certificate and key were set to 'qemu'.
    This user and group exist on compute nodes, but not on controller.
    [1] Error: Could not find group qemu"

    This patch adds 'qemu' user/group on controller node to
    resolve the issue as this user is required to retrieve the cert,
    used by the VNC proxy, the same way as on the compute nodes.

    Change-Id: I3aa774c06d91a3b67726fad0d0ca409cda5b78b9
    Closes-Bug: #1860971

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/707401

[OpenStack Infra (hudson-openstack)]

Change abandoned by Piotr Kopec (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/707401

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 12.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/707401
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=852dc1013d6c18113049468ceb1c1a6178aeb4cf
Submitter: Zuul
Branch: stable/train

commit 852dc1013d6c18113049468ceb1c1a6178aeb4cf
Author: Piotr Kopec <email address hidden>
Date: Tue Nov 19 09:45:05 2019 +0100

    Remove libvirt packaged dependencies

    This re-apply change reverted in I6db56cec954c4d6272548791e6b73bd01b177769
    Problem has been fixed in master and the fix is squashed in this change

    Nova services are now running in the containers but we have still
    a lot of libvirt packages installed on Overcloud systems.
    To delete unnecessary packages on host systems I'm removing following
    dependencies:
    * modifying NovaLibvirtGuests service to run in containers and generate
      config files for libvirt-guests
    * removeing hard dependencies for libvirt-guests service to
      virt-guest-shutdown.target.
      packages.

    Change-Id: I2d0557127f88a492b283897767e57ea126adfe83
    Closes-Bug: 1842932
    (cherry picked from commit 42eb7c98b6d124a4a3c711cf8da217f75a1d1163)

    Create qemu user/group on controller

    Deployment is failing with error [1] because the owner/group
    of the TLS generated certificate and key were set to 'qemu'.
    This user and group exist on compute nodes, but not on controller.
    [1] Error: Could not find group qemu"

    This patch adds 'qemu' user/group on controller node to
    resolve the issue as this user is required to retrieve the cert,
    used by the VNC proxy, the same way as on the compute nodes.

    Change-Id: I3aa774c06d91a3b67726fad0d0ca409cda5b78b9
    Closes-Bug: #1860971
    (cherry picked from commit b8c6154e7afda5525be00189afde15dac649a7b2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.