TLS everywhere timeouts on getcert resubmit

[Grzegorz Grasza]

Description
===========

TLS everywhere brownfield deployment randomly times out on deploying the overcloud. On deeper inspection, there is an issue with the getcert resubmit command, which is run with the '-w' option, waiting on certmonger without a timeout.

/usr/bin/getcert resubmit -i libvirt-vnc-server-cert -f /etc/pki/libvirt-vnc/server-cert.pem -c IPA -N CN=overcloud-novacompute-0.internalapi.ooo.test -K libvirt-vnc/overcloud-novacompute-0.internalapi.ooo.test -D overcloud-novacompute-0.internalapi.ooo.test -C systemctl reload libvirtd -w

There is a repeated message in the logs from certmonger:

Certificate in file "/etc/pki/libvirt-vnc/server-cert.pem" issued by CA and saved, but the CA certificate was not saved.

It turns out that /etc/pki/CA/certs/vnc.crt is a directory instead of a file.

Steps to reproduce
==================

This can happen with different cert files, also mentioned in https://www.ajg.id.au/2019/02/24/applying-tls-everywhere-to-an-existing-openstack-13-queens-cloud/

In my case, the first overcloud deployment finished with an error:

Error response from daemon: error while creating mount source path '/etc/pki/qemu/ca-cert.pem': chown /etc/pki/qemu/ca-cert.pem: no such file or directory.

And with an Ansible timeout the second time.

The underlying issue
=====================

Digging further in, I found out there is an issue with docker volumes, that results in docker daemon creating the directory in place of a path that doesn't exist https://github.com/moby/moby/issues/13121

This happens when new containers are started with new configuration, before certificates are generated and placed in the appropriate paths.

To resolve the issue, we should only mount directories with certificates, or make sure the files mentioned in the volumes configuration are created beforehand.

Using podman wouldn't exhibit this timeout issue - podman returns with an error, if the file doesn't exist.

Hotfix
======

Remove the directories before Ansible times out, restart docker containers.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/672239

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/672239
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=6ca0170eebfad694d6d9e5e0d25f9f1b2fa971ce
Submitter: Zuul
Branch: master

commit 6ca0170eebfad694d6d9e5e0d25f9f1b2fa971ce
Author: Grzegorz Grasza <email address hidden>
Date: Tue Jul 23 11:13:44 2019 +0200

    Check if volume paths exist before executing Docker.

    If the file to be mounted doesn't exist, Docker version 1.13.1
    creates a directory in it's place. This behavior is different from
    Podman's, which throws an error. We want to replicate the Podman
    behavior. Creating directories in place of files causes errors
    which are very difficult to diagnose.

    Related-Bug #1830734

    Change-Id: Ibc458d3badb807a0deef5a799db6682d7128dbc1

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/673548

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/rocky
Review: https://review.opendev.org/673561

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.opendev.org/673565

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673548
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=a85515a89395fbc6c20ff8a0bc4015777750f773
Submitter: Zuul
Branch: stable/stein

commit a85515a89395fbc6c20ff8a0bc4015777750f773
Author: Grzegorz Grasza <email address hidden>
Date: Tue Jul 23 11:13:44 2019 +0200

    Check if volume paths exist before executing Docker.

    If the file to be mounted doesn't exist, Docker version 1.13.1
    creates a directory in it's place. This behavior is different from
    Podman's, which throws an error. We want to replicate the Podman
    behavior. Creating directories in place of files causes errors
    which are very difficult to diagnose.

    Related-Bug #1830734

    Change-Id: Ibc458d3badb807a0deef5a799db6682d7128dbc1
    (cherry picked from commit 6ca0170eebfad694d6d9e5e0d25f9f1b2fa971ce)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673561
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=23154b26d57e4f25456b1f43bc5dff5fb373fc9e
Submitter: Zuul
Branch: stable/rocky

commit 23154b26d57e4f25456b1f43bc5dff5fb373fc9e
Author: Grzegorz Grasza <email address hidden>
Date: Tue Jul 23 11:13:44 2019 +0200

    Check if volume paths exist before executing Docker.

    If the file to be mounted doesn't exist, Docker version 1.13.1
    creates a directory in it's place. This behavior is different from
    Podman's, which throws an error. We want to replicate the Podman
    behavior. Creating directories in place of files causes errors
    which are very difficult to diagnose.

    Depends-On: https://review.opendev.org/675162
    Depends-On: https://review.opendev.org/675262
    Depends-On: https://review.opendev.org/673798
    Depends-On: https://review.opendev.org/673797
    Depends-On: https://review.opendev.org/675666
    Depends-On: https://review.opendev.org/675265

    Related-Bug #1830734

    Change-Id: Ibc458d3badb807a0deef5a799db6682d7128dbc1
    (cherry picked from commit 6ca0170eebfad694d6d9e5e0d25f9f1b2fa971ce)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673565
Committed: https://git.openstack.org/cgit/openstack/paunch/commit/?id=85c1bc9d15acaaf9156991ea57b17d0922329af5
Submitter: Zuul
Branch: stable/queens

commit 85c1bc9d15acaaf9156991ea57b17d0922329af5
Author: Grzegorz Grasza <email address hidden>
Date: Tue Jul 23 11:13:44 2019 +0200

    Check if volume paths exist before executing Docker.

    If the file to be mounted doesn't exist, Docker version 1.13.1
    creates a directory in it's place. This behavior is different from
    Podman's, which throws an error. We want to replicate the Podman
    behavior. Creating directories in place of files causes errors
    which are very difficult to diagnose.

    Depends-On: https://review.opendev.org/675356
    Depends-On: https://review.opendev.org/675357
    Depends-On: https://review.opendev.org/675358
    Depends-On: https://review.opendev.org/675359
    Depends-On: https://review.opendev.org/675360
    Depends-On: https://review.opendev.org/675668
    Related-Bug #1830734

    Change-Id: Ibc458d3badb807a0deef5a799db6682d7128dbc1
    (cherry picked from commit 6ca0170eebfad694d6d9e5e0d25f9f1b2fa971ce)