certmonger does not create cacert files when invoked wityh -F option when doing brownfield TLS

[Ade Lee]

Description
===========
Some services (libvirt for instance) request TLS certs using the -F option for certmonger, which is supposed to save the ca cert in a file. Under certain circumstances, it has been found that while the cert requested is issued and tracked, certmonger fails to create the ca cert file until much later (up to 8 hours). This results in an install failure because the containers that were supposed to start fail to do so because of missing mounts.

This specificallty happens when trying to do a brownfield TLS deployment, on the compute nodes.

After a lot of investigation, it was determined that this happens because certmonger treats the IPA CA as "unrefreshed". The CA is eventually refreshed up to 8 hours later, and the relevant requests are completed and files generated. To make sure the CA is refreshed, certmonger needs to be restarted after the client is registered with IPA. This ensures that the CA is refreshed.

A separate BZ has been filed with IPA to cond-restart certmonger automatically on ipa-client-install.

Steps to reproduce
==================
* Do a TLS brownfield deployment with controllers and compute nodes.

Expected result
===============
Brownfield succeeds.

Actual result
=============
See above

Environment
===========
OSP15+

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691884
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2ed6cf2fdfef3e3e4bd8ab43c1a7906cfe4b931e
Submitter: Zuul
Branch: stable/train

commit 2ed6cf2fdfef3e3e4bd8ab43c1a7906cfe4b931e
Author: Ade Lee <email address hidden>
Date: Fri Oct 18 15:06:02 2019 -0400

    Restart certmnonger after registering system with IPA

    If certmonger is not restarted when the server is registered with
    IPA, then it may define the IPA CA as unreachable. This results
    in CA certs not being stored when cert requests are made with a -F
    option. Eventually, certmonger refreshes itself, but this can
    take up to 8 hours.

    We see this sometimes when doing brownfield deploys. The ca cert
    fails to be created for some requests, resulting in containers
    being unable to load.

    We fix this by simply restarting certmonger after enrollment, and
    avoiding the whole confused state.

    Closes-Bug: 1850647
    Change-Id: Id968a2d5170af1485417e41318e0187d79cd4aae
    (cherry picked from commit bf0bc85ef42bd8d8b3540b1388d64c60e43a8907)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691886
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5eb01c517ff02c7a80c45de44aa47b2aaada2a80
Submitter: Zuul
Branch: stable/stein

commit 5eb01c517ff02c7a80c45de44aa47b2aaada2a80
Author: Ade Lee <email address hidden>
Date: Fri Oct 18 15:06:02 2019 -0400

    Restart certmnonger after registering system with IPA

    If certmonger is not restarted when the server is registered with
    IPA, then it may define the IPA CA as unreachable. This results
    in CA certs not being stored when cert requests are made with a -F
    option. Eventually, certmonger refreshes itself, but this can
    take up to 8 hours.

    We see this sometimes when doing brownfield deploys. The ca cert
    fails to be created for some requests, resulting in containers
    being unable to load.

    We fix this by simply restarting certmonger after enrollment, and
    avoiding the whole confused state.

    Closes-Bug: 1850647
    Change-Id: Id968a2d5170af1485417e41318e0187d79cd4aae
    (cherry picked from commit bf0bc85ef42bd8d8b3540b1388d64c60e43a8907)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691887
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=ebf0a0bd8063f1529a858ec9da3dcf7c09111e19
Submitter: Zuul
Branch: stable/rocky

commit ebf0a0bd8063f1529a858ec9da3dcf7c09111e19
Author: Ade Lee <email address hidden>
Date: Fri Oct 18 15:06:02 2019 -0400

    Restart certmnonger after registering system with IPA

    If certmonger is not restarted when the server is registered with
    IPA, then it may define the IPA CA as unreachable. This results
    in CA certs not being stored when cert requests are made with a -F
    option. Eventually, certmonger refreshes itself, but this can
    take up to 8 hours.

    We see this sometimes when doing brownfield deploys. The ca cert
    fails to be created for some requests, resulting in containers
    being unable to load.

    We fix this by simply restarting certmonger after enrollment, and
    avoiding the whole confused state.

    Closes-Bug: 1850647
    Change-Id: Id968a2d5170af1485417e41318e0187d79cd4aae
    (cherry picked from commit bf0bc85ef42bd8d8b3540b1388d64c60e43a8907)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/691888
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2d77c2a834660d75dfa1da1336b44db69b3cad45
Submitter: Zuul
Branch: stable/queens

commit 2d77c2a834660d75dfa1da1336b44db69b3cad45
Author: Ade Lee <email address hidden>
Date: Fri Oct 18 15:06:02 2019 -0400

    Restart certmnonger after registering system with IPA

    If certmonger is not restarted when the server is registered with
    IPA, then it may define the IPA CA as unreachable. This results
    in CA certs not being stored when cert requests are made with a -F
    option. Eventually, certmonger refreshes itself, but this can
    take up to 8 hours.

    We see this sometimes when doing brownfield deploys. The ca cert
    fails to be created for some requests, resulting in containers
    being unable to load.

    We fix this by simply restarting certmonger after enrollment, and
    avoiding the whole confused state.

    Closes-Bug: 1850647
    Change-Id: Id968a2d5170af1485417e41318e0187d79cd4aae
    (cherry picked from commit bf0bc85ef42bd8d8b3540b1388d64c60e43a8907)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates rocky-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.