yum_update.sh pacemaker_status variable can also match pacemaker_remote nodes

Bug #1688214 reported by Michele Baldessari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Michele Baldessari

Bug Description

As shardy noticed in https://review.openstack.org/#/c/460724, in change I2aae4e2fdfec526c835f8967b54e1db3757bca17 we did the following:
-pacemaker_status=$(systemctl is-active pacemaker || :)
+pacemaker_status=""
+if hiera -c /etc/puppet/hiera.yaml service_names | grep -q pacemaker; then
+ pacemaker_status=$(systemctl is-active pacemaker)
+fi

we did that so due to LP#1668266: we did not want systemctl is-active to fail non non pacemaker nodes. The problem with the above hiera check is that it will match on pacemaker_remote nodes as well. shardy mentioned that we can probably piggyback pacemaker_enabled from hiera. From a quick
test it seems that it returns true on all nodes though:
[root@overcloud-novacompute-0 puppet]# hostname
overcloud-novacompute-0
[root@overcloud-novacompute-0 puppet]# hiera -c /etc/puppet/hiera.yaml pacemaker_enabled
true
[root@overcloud-controller-0 ~]# hostname
overcloud-controller-0
[root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml pacemaker_enabled
true

So we might need to either do a "grep -q \"pacemaker\" or something a bit less relaxed

Changed in tripleo:
importance: Undecided → Medium
assignee: nobody → Michele Baldessari (michele)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (master)

Fix proposed to branch: master
Review: https://review.openstack.org/462467

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
Sofer Athlan-Guyot (sofer-athlan-guyot) wrote :

Hum not sure I get the reach of the problem. In my mind that would mean that the stable/ocata upgrade should fail on minor on non-controller nodes, all the time as non present resource unit should return 3. And if we trigger that where pacemaker is not there then we have return code 3 and it fails.

Am I missing something ?

Adjusting to high before reaching critical or going back to medium :)

Changed in tripleo:
importance: Medium → High
importance: High → Medium
Revision history for this message
Sofer Athlan-Guyot (sofer-athlan-guyot) wrote :

After reading the review it becomes clear that it happen only with pacemaker_remote node type and is not (I think) in the critical way of stable/ocata.

Revision history for this message
Michele Baldessari (michele) wrote :

Agreed, nothing super critical but we best fix it before it bites us :)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (master)

Reviewed: https://review.openstack.org/462467
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2244290424ffa7781fb5b64688908c218cd10ecd
Submitter: Jenkins
Branch: master

commit 2244290424ffa7781fb5b64688908c218cd10ecd
Author: Michele Baldessari <email address hidden>
Date: Thu May 4 11:46:45 2017 +0200

    Fix up pacemaker_status test in yum_update.sh

    In change I2aae4e2fdfec526c835f8967b54e1db3757bca17 we did the
    following:
    -pacemaker_status=$(systemctl is-active pacemaker || :)
    +pacemaker_status=""
    +if hiera -c /etc/puppet/hiera.yaml service_names | grep -q pacemaker;
    then
    + pacemaker_status=$(systemctl is-active pacemaker)
    +fi

    we did that so due to LP#1668266: we did not want systemctl is-active to
    fail on non pacemaker nodes. The problem with the above hiera check is
    that it will match on pacemaker_remote nodes as well.

    We cannot piggyback the pacemaker_enabled hiera key because that is true
    on all nodes. So let's make the test check only for pacemaker service
    without matching pacemaker remote. Tested with:
    1) Test on a controller node with pacemaker service enabled
    [root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml -a service_names |grep '\bpacemaker\b'
    "pacemaker",
    [root@overcloud-controller-0 ~]# echo $?
    0

    2) Test on a compute node without pacemaker:
    [root@overcloud-novacompute-0 puppet]# hiera -c /etc/puppet/hiera.yaml service_names |grep '\bpacemaker\b'
    [root@overcloud-novacompute-0 puppet]# echo $?
    1

    3) Test on a node with pacemaker_remote in the service_names key:
    [root@overcloud-novacompute-0 puppet]# hiera -c /etc/puppet/hiera.yaml service_names |grep '\bpacemaker\b'
    [root@overcloud-novacompute-0 puppet]# echo $?
    1

    [root@overcloud-novacompute-0 puppet]# hiera -c /etc/puppet/hiera.yaml service_names |grep '\bpacemaker_remote\b'
     "pacemaker_remote"]
    [root@overcloud-novacompute-0 puppet]# echo $?
    0

    Change-Id: I54c5756ba6dea791aef89a79bc0b538ba02ae48a
    Closes-Bug: #1688214

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to tripleo-heat-templates (stable/ocata)

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/463278

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to tripleo-heat-templates (stable/ocata)

Reviewed: https://review.openstack.org/463278
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=43b33c7ed8ae2b02dc552767eaa7eb1dec4fb2e1
Submitter: Jenkins
Branch: stable/ocata

commit 43b33c7ed8ae2b02dc552767eaa7eb1dec4fb2e1
Author: Michele Baldessari <email address hidden>
Date: Thu May 4 11:46:45 2017 +0200

    Fix up pacemaker_status test in yum_update.sh

    In change I2aae4e2fdfec526c835f8967b54e1db3757bca17 we did the
    following:
    -pacemaker_status=$(systemctl is-active pacemaker || :)
    +pacemaker_status=""
    +if hiera -c /etc/puppet/hiera.yaml service_names | grep -q pacemaker;
    then
    + pacemaker_status=$(systemctl is-active pacemaker)
    +fi

    we did that so due to LP#1668266: we did not want systemctl is-active to
    fail on non pacemaker nodes. The problem with the above hiera check is
    that it will match on pacemaker_remote nodes as well.

    We cannot piggyback the pacemaker_enabled hiera key because that is true
    on all nodes. So let's make the test check only for pacemaker service
    without matching pacemaker remote. Tested with:
    1) Test on a controller node with pacemaker service enabled
    [root@overcloud-controller-0 ~]# hiera -c /etc/puppet/hiera.yaml -a service_names |grep '\bpacemaker\b'
    "pacemaker",
    [root@overcloud-controller-0 ~]# echo $?
    0

    2) Test on a compute node without pacemaker:
    [root@overcloud-novacompute-0 puppet]# hiera -c /etc/puppet/hiera.yaml service_names |grep '\bpacemaker\b'
    [root@overcloud-novacompute-0 puppet]# echo $?
    1

    3) Test on a node with pacemaker_remote in the service_names key:
    [root@overcloud-novacompute-0 puppet]# hiera -c /etc/puppet/hiera.yaml service_names |grep '\bpacemaker\b'
    [root@overcloud-novacompute-0 puppet]# echo $?
    1

    [root@overcloud-novacompute-0 puppet]# hiera -c /etc/puppet/hiera.yaml service_names |grep '\bpacemaker_remote\b'
     "pacemaker_remote"]
    [root@overcloud-novacompute-0 puppet]# echo $?
    0

    NB: cherry-pick was not 100% clean due to unrelated lines being cleaned
    up in master.

    Change-Id: I54c5756ba6dea791aef89a79bc0b538ba02ae48a
    Closes-Bug: #1688214
    (cherry picked from commit 2244290424ffa7781fb5b64688908c218cd10ecd)

tags: added: in-stable-ocata
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 7.0.0.0b2

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0b2 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/tripleo-heat-templates 6.2.0

This issue was fixed in the openstack/tripleo-heat-templates 6.2.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.