Compute node upgrade from queens -> train with [qemu-nbd-client-cert]: Could not evaluate: Could not get certificate

Bug #1886047 reported by Grzegorz Grasza
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Emilien Macchi

Bug Description

TLS Everywhere environment. Compute node upgrade from 13 -> 16 fails with below error. This is when running "openstack overcloud upgrade run --stack overcloud --limit compute-1" after a successful LAPP from RHEL7 -> RHEL8

Error: /Stage[main]/Tripleo::Profile::Base::Certmonger_user/Tripleo::Certmonger::Qemu[qemu-nbd-client-cert]/Certmonger_certificate[qemu-nbd-client-cert]: Could not evaluate: Could not get certificate: Server at https://ipa-server.redhat.local/ipa/xml denied our request, giving up: 2100 (RPC failed at server. Insufficient access: Insufficient 'add' privilege to add the entry '<email address hidden>,cn=services,cn=accounts,dc=redhat,dc=local'.).\

On further research it turns out that novajoin has guest as transport endpoint user instead of the <hash> that it should use.

novajoin notifier service failed to start with

ERROR join File "/usr/lib/python3.6/site-packages/amqp/method_framing.py", line 55, in on_frame
ERROR join callback(channel, method_sig, buf, None)
ERROR join File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 510, in on_inbound_method
ERROR join method_sig, payload, content,
ERROR join File "/usr/lib/python3.6/site-packages/amqp/abstract_channel.py", line 126, in dispatch_method
ERROR join listener(*args)
ERROR join File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 639, in _on_close
ERROR join (class_id, method_id), ConnectionError)
ERROR join amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.

novajoin_notify service couldn't connect to amqp and uses patch /var/lib/config-data/... instead of /var/lib/config-data/puppet-generated...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/739037

Changed in tripleo:
assignee: nobody → Grzegorz Grasza (xek)
status: New → In Progress
Changed in tripleo:
milestone: none → victoria-1
importance: Undecided → High
tags: added: queens-backport-potential train-backport-potential ussuri-backport-potential
Changed in tripleo:
assignee: Grzegorz Grasza (xek) → Emilien Macchi (emilienm)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/739037
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=53900ae3a9c15cfd6fe283bcb5baba8c21f2f70b
Submitter: Zuul
Branch: master

commit 53900ae3a9c15cfd6fe283bcb5baba8c21f2f70b
Author: Emilien Macchi <email address hidden>
Date: Mon Jul 6 11:55:24 2020 -0400

    Fix bind mount volumes for novajoin containers

    /var/lib/config-data/novajoin should not be bind mounted directly.
    The way it works, is that a step 1 we generate all the config files with
    Puppet and put the /etc/novajoin content into
    /var/lib/config-data/novajoin and then filter out that is actually
    needed by the containers into
    /var/lib/config-data/puppet-generated/novajoin.

    /var/lib/config-data/puppet-generated/novajoin will have
    /var/lib/config-data/puppet-generated/novajoin/etc/novajoin with the
    config files but it won't have the files excluded by container-puppet.sh
    (e.g. host files, some CA/PKI files, etc).

    In the container configs, we want to bind mount
    /var/lib/config-data/puppet-generated/novajoin into
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    and when the container will start, Kolla will automatically copy the
    content of
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    into / by deep merging; so /etc/novajoin will have the config files.

    Closes-Bug: #1886047
    Resolves: rhbz#1853268

    Change-Id: I9d28b5f7f40d43e017153cb274a3f419e23511c3

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/740623

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/740624

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/740624
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=c276a7e307be193ae74fee2bc2b259200cd008d1
Submitter: Zuul
Branch: stable/train

commit c276a7e307be193ae74fee2bc2b259200cd008d1
Author: Emilien Macchi <email address hidden>
Date: Mon Jul 6 11:55:24 2020 -0400

    Fix bind mount volumes for novajoin containers

    /var/lib/config-data/novajoin should not be bind mounted directly.
    The way it works, is that a step 1 we generate all the config files with
    Puppet and put the /etc/novajoin content into
    /var/lib/config-data/novajoin and then filter out that is actually
    needed by the containers into
    /var/lib/config-data/puppet-generated/novajoin.

    /var/lib/config-data/puppet-generated/novajoin will have
    /var/lib/config-data/puppet-generated/novajoin/etc/novajoin with the
    config files but it won't have the files excluded by container-puppet.sh
    (e.g. host files, some CA/PKI files, etc).

    In the container configs, we want to bind mount
    /var/lib/config-data/puppet-generated/novajoin into
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    and when the container will start, Kolla will automatically copy the
    content of
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    into / by deep merging; so /etc/novajoin will have the config files.

    Closes-Bug: #1886047
    Resolves: rhbz#1853268

    Change-Id: I9d28b5f7f40d43e017153cb274a3f419e23511c3
    (cherry picked from commit 53900ae3a9c15cfd6fe283bcb5baba8c21f2f70b)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/ussuri)

Reviewed: https://review.opendev.org/740623
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=03ab2f26d9c3b4971cea3d9cf1cfb502f6c745e9
Submitter: Zuul
Branch: stable/ussuri

commit 03ab2f26d9c3b4971cea3d9cf1cfb502f6c745e9
Author: Emilien Macchi <email address hidden>
Date: Mon Jul 6 11:55:24 2020 -0400

    Fix bind mount volumes for novajoin containers

    /var/lib/config-data/novajoin should not be bind mounted directly.
    The way it works, is that a step 1 we generate all the config files with
    Puppet and put the /etc/novajoin content into
    /var/lib/config-data/novajoin and then filter out that is actually
    needed by the containers into
    /var/lib/config-data/puppet-generated/novajoin.

    /var/lib/config-data/puppet-generated/novajoin will have
    /var/lib/config-data/puppet-generated/novajoin/etc/novajoin with the
    config files but it won't have the files excluded by container-puppet.sh
    (e.g. host files, some CA/PKI files, etc).

    In the container configs, we want to bind mount
    /var/lib/config-data/puppet-generated/novajoin into
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    and when the container will start, Kolla will automatically copy the
    content of
    /var/lib/config-data/puppet-generated/novajoin:/var/lib/kolla/config_files/src
    into / by deep merging; so /etc/novajoin will have the config files.

    Closes-Bug: #1886047
    Resolves: rhbz#1853268

    Change-Id: I9d28b5f7f40d43e017153cb274a3f419e23511c3
    (cherry picked from commit 53900ae3a9c15cfd6fe283bcb5baba8c21f2f70b)

tags: added: in-stable-ussuri
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.4.0

This issue was fixed in the openstack/tripleo-heat-templates 11.4.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.