pacemaker_restart_bundle.sh fail when TRIPLEO_MINOR_UPDATE is undefined

Bug #1849503 reported by Damien Ciabrini
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
High
Damien Ciabrini

Bug Description

Since I92d4ddf2feeac06ce14468ae928c283f3fd04f45, HA containers <service>_restart_bundle.sh run script pacemaker_restart_bundle.sh to drive the resource restart behaviour based on the value of environment variable TRIPLEO_MINOR_UPDATE. This allows the behaviour to be adapted whether we're running a stack update or a minor update.

However the script doesn't work as expected when TRIPLEO_MINOR_UPDATE is undefined, due to pacemaker_restart_bundle.sh using "set -u". Consequently, any stack update that would trigger that container would error out with e.g.:

 paunch --debug apply --file /var/lib/tripleo-config/hashed-docker-container-startup-config-step_2.json --config-id tripleo_step2 --managed-by tripleo-Controller 2>&1 | tee /tmp/restart
Error running ['docker', 'run', '--name', 'haproxy_restart_bundle', '--label', 'config_id=tripleo_step2', '--label', 'container_name=haproxy_restart_bundle', '--label', 'managed_by=paunch', '--label', 'config_data={"start_order": 2, "image
": "192.168.24.1:8787/rhosp13/openstack-haproxy:2019-10-09.1", "environment": ["TRIPLEO_MINOR_UPDATE", "TRIPLEO_CONFIG_HASH=358ed8a61f99e0575302387a156889ee"], "config_volume": "haproxy", "command": "/pacemaker_restart_bundle.sh haproxy-bu
ndle haproxy", "user": "root", "volumes": ["/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro"
, "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/var/lib/
docker-config-scripts/pacemaker_restart_bundle.sh:/pacemaker_restart_bundle.sh:ro", "/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro", "/dev/shm:/dev/shm:rw", "/etc/puppet:/etc/puppet:ro", "/var/lib/config-data/puppet-generated/
haproxy/:/var/lib/kolla/config_files/src:ro"], "net": "host", "detach": false}', '--env=TRIPLEO_MINOR_UPDATE', '--env=TRIPLEO_CONFIG_HASH=358ed8a61f99e0575302387a156889ee', '--net=host', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro',
'--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt
:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/var/lib
/docker-config-scripts/pacemaker_restart_bundle.sh:/pacemaker_restart_bundle.sh:ro', '--volume=/etc/corosync/corosync.conf:/etc/corosync/corosync.conf:ro', '--volume=/dev/shm:/dev/shm:rw', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/
var/lib/config-data/puppet-generated/haproxy/:/var/lib/kolla/config_files/src:ro', '192.168.24.1:8787/rhosp13/openstack-haproxy:2019-10-09.1', '/pacemaker_restart_bundle.sh', 'haproxy-bundle', 'haproxy']. [1]

stdout: Bundle: haproxy-bundle
  Docker: image=192.168.24.1:8787/rhosp13/openstack-haproxy:pcmklatest network=host options="--user=root --log-driver=journald -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" replicas=3 run-command="/bin/bash /usr/local/bin/kolla_start"
  Storage Mapping:
   options=ro source-dir=/var/lib/kolla/config_files/haproxy.json target-dir=/var/lib/kolla/config_files/config.json (haproxy-cfg-files)
   options=ro source-dir=/var/lib/config-data/puppet-generated/haproxy/ target-dir=/var/lib/kolla/config_files/src (haproxy-cfg-data)
   options=ro source-dir=/etc/hosts target-dir=/etc/hosts (haproxy-hosts)
   options=ro source-dir=/etc/localtime target-dir=/etc/localtime (haproxy-localtime)
   options=rw source-dir=/var/lib/haproxy target-dir=/var/lib/haproxy (haproxy-var-lib)
   options=ro source-dir=/etc/pki/ca-trust/extracted target-dir=/etc/pki/ca-trust/extracted (haproxy-pki-extracted)
   options=ro source-dir=/etc/pki/tls/certs/ca-bundle.crt target-dir=/etc/pki/tls/certs/ca-bundle.crt (haproxy-pki-ca-bundle-crt)
   options=ro source-dir=/etc/pki/tls/certs/ca-bundle.trust.crt target-dir=/etc/pki/tls/certs/ca-bundle.trust.crt (haproxy-pki-ca-bundle-trust-crt)
   options=ro source-dir=/etc/pki/tls/cert.pem target-dir=/etc/pki/tls/cert.pem (haproxy-pki-cert)
   options=rw source-dir=/dev/log target-dir=/dev/log (haproxy-dev-log)
   options=ro source-dir=/etc/pki/tls/private/overcloud_endpoint.pem target-dir=/var/lib/kolla/config_files/src-tls/etc/pki/tls/private/overcloud_endpoint.pem (haproxy-cert)

stderr: /pacemaker_restart_bundle.sh: line 11: TRIPLEO_MINOR_UPDATE: unbound variable

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.opendev.org/690600

Changed in tripleo:
milestone: none → ussuri-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/690880

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/690890

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/690904

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/690916

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.opendev.org/690600
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=81610bdc366527d946304014bcd1782cf0a858d8
Submitter: Zuul
Branch: master

commit 81610bdc366527d946304014bcd1782cf0a858d8
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 23 16:56:38 2019 +0200

    HA: ensure TRIPLEO_MINOR_UPDATE is defined for <svc>_restart_bundle

    Containers <svc>_restart_bundle use script pacemaker_restart_bundle.sh
    which behaves according to the value of environment variable
    TRIPLEO_MINOR_UPDATE. Set a default value in case this variable is
    unset (i.e. during stack update).

    Change-Id: I59da2d3c50fa30a8f3e557a16367f889b103a6f8
    Closes-Bug: #1849503

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/train)

Reviewed: https://review.opendev.org/690880
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=19693b711978ad03144fecd2a06249b9117fcc26
Submitter: Zuul
Branch: stable/train

commit 19693b711978ad03144fecd2a06249b9117fcc26
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 23 16:56:38 2019 +0200

    HA: ensure TRIPLEO_MINOR_UPDATE is defined for <svc>_restart_bundle

    Containers <svc>_restart_bundle use script pacemaker_restart_bundle.sh
    which behaves according to the value of environment variable
    TRIPLEO_MINOR_UPDATE. Set a default value in case this variable is
    unset (i.e. during stack update).

    Change-Id: I59da2d3c50fa30a8f3e557a16367f889b103a6f8
    Closes-Bug: #1849503
    (cherry picked from commit 81610bdc366527d946304014bcd1782cf0a858d8)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/stein)

Reviewed: https://review.opendev.org/690890
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=9770ed2680c0c51067600ca2bdc11ca0371f42b4
Submitter: Zuul
Branch: stable/stein

commit 9770ed2680c0c51067600ca2bdc11ca0371f42b4
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 23 16:56:38 2019 +0200

    HA: ensure TRIPLEO_MINOR_UPDATE is defined for <svc>_restart_bundle

    Containers <svc>_restart_bundle use script pacemaker_restart_bundle.sh
    which behaves according to the value of environment variable
    TRIPLEO_MINOR_UPDATE. Set a default value in case this variable is
    unset (i.e. during stack update).

    Change-Id: I59da2d3c50fa30a8f3e557a16367f889b103a6f8
    Closes-Bug: #1849503
    (cherry picked from commit 81610bdc366527d946304014bcd1782cf0a858d8)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/rocky)

Reviewed: https://review.opendev.org/690904
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=a9b558f3dd5ed68a1e24d750616d7dec811864f6
Submitter: Zuul
Branch: stable/rocky

commit a9b558f3dd5ed68a1e24d750616d7dec811864f6
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 23 16:56:38 2019 +0200

    HA: ensure TRIPLEO_MINOR_UPDATE is defined for <svc>_restart_bundle

    Containers <svc>_restart_bundle use script pacemaker_restart_bundle.sh
    which behaves according to the value of environment variable
    TRIPLEO_MINOR_UPDATE. Set a default value in case this variable is
    unset (i.e. during stack update).

    Change-Id: I59da2d3c50fa30a8f3e557a16367f889b103a6f8
    Closes-Bug: #1849503
    (cherry picked from commit 81610bdc366527d946304014bcd1782cf0a858d8)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/queens)

Reviewed: https://review.opendev.org/690916
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=55de8db0102cf48d0422007293e56ad677d3e03c
Submitter: Zuul
Branch: stable/queens

commit 55de8db0102cf48d0422007293e56ad677d3e03c
Author: Damien Ciabrini <email address hidden>
Date: Wed Oct 23 16:56:38 2019 +0200

    HA: ensure TRIPLEO_MINOR_UPDATE is defined for <svc>_restart_bundle

    Containers <svc>_restart_bundle use script pacemaker_restart_bundle.sh
    which behaves according to the value of environment variable
    TRIPLEO_MINOR_UPDATE. Set a default value in case this variable is
    unset (i.e. during stack update).

    Change-Id: I59da2d3c50fa30a8f3e557a16367f889b103a6f8
    Closes-Bug: #1849503
    (cherry picked from commit 81610bdc366527d946304014bcd1782cf0a858d8)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 12.0.0

This issue was fixed in the openstack/tripleo-heat-templates 12.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 10.6.2

This issue was fixed in the openstack/tripleo-heat-templates 10.6.2 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 11.3.1

This issue was fixed in the openstack/tripleo-heat-templates 11.3.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates rocky-eol

This issue was fixed in the openstack/tripleo-heat-templates rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates queens-eol

This issue was fixed in the openstack/tripleo-heat-templates queens-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.