haproxy-bundle double mount with tls-everywhere

[Michele Baldessari]

Seen via https://bugzilla.redhat.com/show_bug.cgi?id=1757979:
        "Error running ['podman', 'run', '--name', 'haproxy_init_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=haproxy_init_bundle', '--label', 'managed_by=paunch'
, '--label', 'config_data={\"command\": [\"/container_puppet_apply.sh\", \"2\", \"file,file_line,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ip,pacemake
r::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation\", \"include ::tripleo::profile::base::pacemaker; include ::tripleo::profile::pacemaker::haproxy_bundle\", \"--
debug\"], \"detach\": false, \"environment\": [\"TRIPLEO_DEPLOY_IDENTIFIER=1570035685\"], \"image\": \"192.168.24.1:8787/rhosp16/openstack-haproxy:20190927.1\", \"ipc\": \"host\", \"net\": \"
host\", \"privileged\": true, \"start_order\": 3, \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/
ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/cer
ts/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro\", \"/var/l
ib/container-config-scripts/container_puppet_apply.sh:/container_puppet_apply.sh:ro\", \"/etc/puppet:/tmp/puppet-etc:ro\", \"/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/mo
dules:ro\", \"/etc/pki/tls/private/overcloud_endpoint.pem:/etc/pki/tls/private/overcloud_endpoint.pem:ro\", \"/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro\", \"/etc/pki/tls/private/haproxy:/etc/pki/tls
/private/haproxy:ro\", \"/etc/pki/tls/certs/haproxy:/etc/pki/tls/certs/haproxy:ro\"]}', '--conmon-pidfile=/var/run/haproxy_init_bundle.pid', '--log-driver', 'k8s-file', '--log-opt', 'path=/va
r/log/containers/stdouts/haproxy_init_bundle.log', '--env=TRIPLEO_DEPLOY_IDENTIFIER=1570035685', '--net=host', '--ipc=host', '--privileged=true', '--user=root', '--volume=/etc/hosts:/etc/host
s:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/
anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--v
olume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro', '--volume=/var/lib/container-config-scripts/container_puppe
t_apply.sh:/container_puppet_apply.sh:ro', '--volume=/etc/puppet:/tmp/puppet-etc:ro', '--volume=/usr/share/openstack-puppet/modules:/usr/share/openstack-puppet/modules:ro', '--volume=/etc/pki
/tls/private/overcloud_endpoint.pem:/etc/pki/tls/private/overcloud_endpoint.pem:ro', '--volume=/etc/ipa/ca.crt:/etc/ipa/ca.crt:ro', '--volume=/etc/pki/tls/private/haproxy:/etc/pki/tls/private
/haproxy:ro', '--volume=/etc/pki/tls/certs/haproxy:/etc/pki/tls/certs/haproxy:ro', '192.168.24.1:8787/rhosp16/openstack-haproxy:20190927.1', '/container_puppet_apply.sh', '2', 'file,file_line
,concat,augeas,pacemaker::resource::bundle,pacemaker::property,pacemaker::resource::ip,pacemaker::resource::ocf,pacemaker::constraint::order,pacemaker::constraint::colocation', 'include ::tri
pleo::profile::base::pacemaker; include ::tripleo::profile::pacemaker::haproxy_bundle', '--debug']. [125]",
        "stdout: ",
        "stderr: Error: /etc/ipa/ca.crt: duplicate mount destination",

This is due more recent podman commit:
https://github.com/containers/libpod/commit/9ee50fe2c7d31e5a6209b63f7735a965dc204131

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/686399

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/688055

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686399
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=668cc684fc93503128c946dae9f331b49437ec2c
Submitter: Zuul
Branch: master

commit 668cc684fc93503128c946dae9f331b49437ec2c
Author: Michele Baldessari <email address hidden>
Date: Thu Oct 3 15:04:47 2019 +0200

    Fix double cert mount in haproxy bundle when using tls everywhere

    When deploying with tls-everywhere the haproxy_init_bundle container
    will bind mount /etc/ipa/ca.crt twice and starting with podman 1.{3,4}.x
    this will break:

      "stderr: Error: /etc/ipa/ca.crt: duplicate mount destination"

    The TLS bind mounts are needed in haproxy_init_bundle (which is only
    in charge of creating the pcmk resource) because puppet-haproxy uses
    a validate command to check the config which will fail if we simply
    remove those bind mounts. Instead of skipping this verification with
    some parameter/tag, we try and avoid the duplicate cert by removing the
    ca.crt bind mount from deployed_cert_mounts.

    The duplication comes from:
    1) https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/containers-common.yaml#L122-L127
    2) https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/haproxy/haproxy-pacemaker-puppet.yaml#L263

    Since changing it into 1) has large implication, we just avoid
    redefining it in 2).

    Tested with a full tls-everywhere deploy with the applied patch and the
    error is not seen any longer.

    Change-Id: I6493fd090c808da01d19cc12d1b8371c67708904
    Related-Bug: #1833347
    Closes-Bug: #1846495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/688055
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7e303fdbbb580db2769375c31088f4ba583bc00e
Submitter: Zuul
Branch: stable/stein

commit 7e303fdbbb580db2769375c31088f4ba583bc00e
Author: Michele Baldessari <email address hidden>
Date: Thu Oct 3 15:04:47 2019 +0200

    Fix double cert mount in haproxy bundle when using tls everywhere

    When deploying with tls-everywhere the haproxy_init_bundle container
    will bind mount /etc/ipa/ca.crt twice and starting with podman 1.{3,4}.x
    this will break:

      "stderr: Error: /etc/ipa/ca.crt: duplicate mount destination"

    The TLS bind mounts are needed in haproxy_init_bundle (which is only
    in charge of creating the pcmk resource) because puppet-haproxy uses
    a validate command to check the config which will fail if we simply
    remove those bind mounts. Instead of skipping this verification with
    some parameter/tag, we try and avoid the duplicate cert by removing the
    ca.crt bind mount from deployed_cert_mounts.

    The duplication comes from:
    1) https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/containers-common.yaml#L122-L127
    2) https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/haproxy/haproxy-pacemaker-puppet.yaml#L263

    Since changing it into 1) has large implication, we just avoid
    redefining it in 2).

    Tested with a full tls-everywhere deploy with the applied patch and the
    error is not seen any longer.

    (cherry picked from commit 668cc684fc93503128c946dae9f331b49437ec2c)

    Change-Id: I6493fd090c808da01d19cc12d1b8371c67708904
    Related-Bug: #1833347
    Closes-Bug: #1846495

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.2 release.