Compute node upgrade from queens -> train with [qemu-nbd-client-cert]: Could not evaluate: Could not get certificate

[Grzegorz Grasza]

TLS Everywhere environment. Compute node upgrade from 13 -> 16 fails with below error. This is when running "openstack overcloud upgrade run --stack overcloud --limit compute-1" after a successful LAPP from RHEL7 -> RHEL8

Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/Certmonger_certificate[qemu-nbd-client-cert]: Could not evaluate: Could not get certificate: Server at https://ipa-server.redhat.local/ipa/xml denied our request, giving up: 2100 (RPC failed at server. Insufficient access: Insufficient 'add' privilege to add the entry '<email address hidden>,cn=services,cn=accounts,dc=redhat,dc=local'.).\

On further research it turns out that novajoin has guest as transport endpoint user instead of the <hash> that it should use.

novajoin notifier service failed to start with

ERROR join File "/usr/lib/python3.6/site-packages/amqp/method_framing.py", line 55, in on_frame
ERROR join callback(channel, method_sig, buf, None)
ERROR join File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 510, in on_inbound_method
ERROR join method_sig, payload, content,
ERROR join File "/usr/lib/python3.6/site-packages/amqp/abstract_channel.py", line 126, in dispatch_method
ERROR join listener(*args)
ERROR join File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 639, in _on_close
ERROR join (class_id, method_id), ConnectionError)
ERROR join amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.

novajoin_notify service couldn't connect to amqp and uses patch /var/lib/config-data/... instead of /var/lib/config-data/puppet-generated...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/739037

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/739037
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=53900ae3a9c15cfd6fe283bcb5baba8c21f2f70b
Submitter: Zuul
Branch: master

commit 53900ae3a9c15cfd6fe283bcb5baba8c21f2f70b
Author: Emilien Macchi <email address hidden>
Date: Mon Jul 6 11:55:24 2020 -0400

    Fix bind mount volumes for novajoin containers

    /var/lib/config-data/novajoin should not be bind mounted directly.
    The way it works, is that a step 1 we generate all the config files with
    Puppet and put the /etc/novajoin content into
    /var/lib/config-data/novajoin and then filter out that is actually
    needed by the containers into
    /var/lib/config-data/puppet-generated/novajoin.

    /var/lib/config-data/puppet-generated/novajoin will have
    /var/lib/config-data/puppet-generated/novajoin/etc/novajoin with the
    config files but it won't have the files excluded by container-puppet.sh
    (e.g. host files, some CA/PKI files, etc).

    In the container configs, we want to bind mount
    /var/lib/config-data/puppet-generated/novajoin into
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    and when the container will start, Kolla will automatically copy the
    content of
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    into / by deep merging; so /etc/novajoin will have the config files.

    Closes-Bug: #1886047
    Resolves: rhbz#1853268

    Change-Id: I9d28b5f7f40d43e017153cb274a3f419e23511c3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/740623

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/740624

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740624
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c276a7e307be193ae74fee2bc2b259200cd008d1
Submitter: Zuul
Branch: stable/train

commit c276a7e307be193ae74fee2bc2b259200cd008d1
Author: Emilien Macchi <email address hidden>
Date: Mon Jul 6 11:55:24 2020 -0400

    Fix bind mount volumes for novajoin containers

    /var/lib/config-data/novajoin should not be bind mounted directly.
    The way it works, is that a step 1 we generate all the config files with
    Puppet and put the /etc/novajoin content into
    /var/lib/config-data/novajoin and then filter out that is actually
    needed by the containers into
    /var/lib/config-data/puppet-generated/novajoin.

    /var/lib/config-data/puppet-generated/novajoin will have
    /var/lib/config-data/puppet-generated/novajoin/etc/novajoin with the
    config files but it won't have the files excluded by container-puppet.sh
    (e.g. host files, some CA/PKI files, etc).

    In the container configs, we want to bind mount
    /var/lib/config-data/puppet-generated/novajoin into
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    and when the container will start, Kolla will automatically copy the
    content of
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    into / by deep merging; so /etc/novajoin will have the config files.

    Closes-Bug: #1886047
    Resolves: rhbz#1853268

    Change-Id: I9d28b5f7f40d43e017153cb274a3f419e23511c3
    (cherry picked from commit 53900ae3a9c15cfd6fe283bcb5baba8c21f2f70b)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/740623
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=03ab2f26d9c3b4971cea3d9cf1cfb502f6c745e9
Submitter: Zuul
Branch: stable/ussuri

commit 03ab2f26d9c3b4971cea3d9cf1cfb502f6c745e9
Author: Emilien Macchi <email address hidden>
Date: Mon Jul 6 11:55:24 2020 -0400

    Fix bind mount volumes for novajoin containers

    /var/lib/config-data/novajoin should not be bind mounted directly.
    The way it works, is that a step 1 we generate all the config files with
    Puppet and put the /etc/novajoin content into
    /var/lib/config-data/novajoin and then filter out that is actually
    needed by the containers into
    /var/lib/config-data/puppet-generated/novajoin.

    /var/lib/config-data/puppet-generated/novajoin will have
    /var/lib/config-data/puppet-generated/novajoin/etc/novajoin with the
    config files but it won't have the files excluded by container-puppet.sh
    (e.g. host files, some CA/PKI files, etc).

    In the container configs, we want to bind mount
    /var/lib/config-data/puppet-generated/novajoin into
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    and when the container will start, Kolla will automatically copy the
    content of
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    into / by deep merging; so /etc/novajoin will have the config files.

    Closes-Bug: #1886047
    Resolves: rhbz#1853268

    Change-Id: I9d28b5f7f40d43e017153cb274a3f419e23511c3
    (cherry picked from commit 53900ae3a9c15cfd6fe283bcb5baba8c21f2f70b)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.