Undercloud deploy fails with Error: /etc/puppet: duplicate mount destination

[Rabi Mishra]

Noticed at:

https://logs.rdoproject.org/11/666211/1/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001/e39cf1b/logs/undercloud/home/zuul/undercloud_install.log.txt.gz

It seems like we're using the same mount multiple times in neutron_ovs_bridge.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/666220

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/666222

[yatin (yatinkarel)]

This is hit after podman-1.3.2 is pushed in centos-extras recently and is affecting most of the jobs both in Check and promotion pipeline[1]. Added promotion-blocker tag. There is work going on to include podman-1.4.0 in RDO [2] but it has some other issues so fixing it altogether will take some time. So let's see if https://review.opendev.org/666220 avoids the issue with podman-1.3 completely, i hope it will fix it considering standalone jobs are passing(which covers non-ovs scenarios) else we can go with excluding podman-1.3 from extras repo by https://review.opendev.org/#/c/666222/ in meanwhile until a good version of podman is pushed.

[1] https://trunk-primary.rdoproject.org/api-centos-master-uc/api/civotes_detail.html?commit_hash=72000c0a037ed187c7294cff48f6bcea48bb781f&distro_hash=af06cae057e85e2857d249cea5fbccb92a24e47a
[2] https://review.rdoproject.org/r/#/c/21180/

[yatin (yatinkarel)]

current status, https://review.opendev.org/666220 is good and working, but it's taking time to merge due to long zuul queue and random failures, to unblock CI jobs in master and stein we have temporary patched tripleo-heat-templates package in RDO with https://review.rdoproject.org/r/#/c/21191/ and https://review.rdoproject.org/r/#/c/21193/, RDO patches can be reverted once https://review.opendev.org/666220 and it's cherry-pick in stein merges.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666220
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=b6c88a99aaefb5b89ec04f4c85e131d559c575bc
Submitter: Zuul
Branch: master

commit b6c88a99aaefb5b89ec04f4c85e131d559c575bc
Author: Rabi Mishra <email address hidden>
Date: Wed Jun 19 10:37:36 2019 +0530

    Remove duplicate mount destinations in containers

    Change-Id: Ibbb4548f4cd5ea72a20bc1b0b7073cfb14be2068
    Closes-Bug: #1833347

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/666436

[OpenStack Infra (hudson-openstack)]

Change abandoned by yatin (<email address hidden>) on branch: master
Review: https://review.opendev.org/666222
Reason: https://review.opendev.org/#/c/666220/ merged and is cherry-picked to stein:- https://review.opendev.org/#/c/666220/, this is no longer needed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/666436
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=aa280e94ca74e332b0c11e18f5c472a8f67bdffe
Submitter: Zuul
Branch: stable/stein

commit aa280e94ca74e332b0c11e18f5c472a8f67bdffe
Author: Rabi Mishra <email address hidden>
Date: Wed Jun 19 10:37:36 2019 +0530

    Remove duplicate mount destinations in containers

    Change-Id: Ibbb4548f4cd5ea72a20bc1b0b7073cfb14be2068
    Closes-Bug: #1833347
    (cherry picked from commit b6c88a99aaefb5b89ec04f4c85e131d559c575bc)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 10.6.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 11.1.0 release.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/686399

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/stein
Review: https://review.opendev.org/688055

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686399
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=668cc684fc93503128c946dae9f331b49437ec2c
Submitter: Zuul
Branch: master

commit 668cc684fc93503128c946dae9f331b49437ec2c
Author: Michele Baldessari <email address hidden>
Date: Thu Oct 3 15:04:47 2019 +0200

    Fix double cert mount in haproxy bundle when using tls everywhere

    When deploying with tls-everywhere the haproxy_init_bundle container
    will bind mount /etc/ipa/ca.crt twice and starting with podman 1.{3,4}.x
    this will break:

      "stderr: Error: /etc/ipa/ca.crt: duplicate mount destination"

    The TLS bind mounts are needed in haproxy_init_bundle (which is only
    in charge of creating the pcmk resource) because puppet-haproxy uses
    a validate command to check the config which will fail if we simply
    remove those bind mounts. Instead of skipping this verification with
    some parameter/tag, we try and avoid the duplicate cert by removing the
    ca.crt bind mount from deployed_cert_mounts.

    The duplication comes from:
    1) https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/containers-common.yaml#L122-L127
    2) https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/haproxy/haproxy-pacemaker-puppet.yaml#L263

    Since changing it into 1) has large implication, we just avoid
    redefining it in 2).

    Tested with a full tls-everywhere deploy with the applied patch and the
    error is not seen any longer.

    Change-Id: I6493fd090c808da01d19cc12d1b8371c67708904
    Related-Bug: #1833347
    Closes-Bug: #1846495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/688055
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=7e303fdbbb580db2769375c31088f4ba583bc00e
Submitter: Zuul
Branch: stable/stein

commit 7e303fdbbb580db2769375c31088f4ba583bc00e
Author: Michele Baldessari <email address hidden>
Date: Thu Oct 3 15:04:47 2019 +0200

    Fix double cert mount in haproxy bundle when using tls everywhere

    When deploying with tls-everywhere the haproxy_init_bundle container
    will bind mount /etc/ipa/ca.crt twice and starting with podman 1.{3,4}.x
    this will break:

      "stderr: Error: /etc/ipa/ca.crt: duplicate mount destination"

    The TLS bind mounts are needed in haproxy_init_bundle (which is only
    in charge of creating the pcmk resource) because puppet-haproxy uses
    a validate command to check the config which will fail if we simply
    remove those bind mounts. Instead of skipping this verification with
    some parameter/tag, we try and avoid the duplicate cert by removing the
    ca.crt bind mount from deployed_cert_mounts.

    The duplication comes from:
    1) https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/containers-common.yaml#L122-L127
    2) https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/haproxy/haproxy-pacemaker-puppet.yaml#L263

    Since changing it into 1) has large implication, we just avoid
    redefining it in 2).

    Tested with a full tls-everywhere deploy with the applied patch and the
    error is not seen any longer.

    (cherry picked from commit 668cc684fc93503128c946dae9f331b49437ec2c)

    Change-Id: I6493fd090c808da01d19cc12d1b8371c67708904
    Related-Bug: #1833347
    Closes-Bug: #1846495