stonith level do not explicitely require stonith resources

Bug #1866214 reported by Michele Baldessari
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Medium
Michele Baldessari

Bug Description

Currently stonith levels kind of work out of pure luck. If puppet decides to reorder resources stonith levels can fail with:
            "Error: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200305-60559-tn9ici create failed: Error: Stonith resource(s) 'stonith-fence_ipmilan-525400e05b23' do not exist, use --force to override",
            "Error: /Stage[main]/Tripleo::Fencing/Pacemaker::Stonith::Level[stonith-1-525400e05b23]/Pcmk_stonith_level[stonith-level-1-$(/usr/sbin/crm_node -n)-stonith-fence_ipmilan-525400e05b23]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200305-60559-tn9ici create failed: Error: Stonith resource(s) 'stonith-fence_ipmilan-525400e05b23' do not exist, use --force to override"

And currently nowhere in the code do we mandate that the single stonith resources need to be created *before* the stonith levels which make use of them.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (master)

Fix proposed to branch: master
Review: https://review.opendev.org/711527

Changed in tripleo:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (master)

Reviewed: https://review.opendev.org/711527
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=b706f5365a84712eea8a133a9b8029878dcec2d2
Submitter: Zuul
Branch: master

commit b706f5365a84712eea8a133a9b8029878dcec2d2
Author: Michele Baldessari <email address hidden>
Date: Thu Mar 5 19:03:00 2020 +0100

    Make sure we create stonith resources before stonith levels

    Currently stonith levels kind of work out of pure luck. If puppet
    decides to reorder resources stonith levels can fail with:

                "Error: /Stage[main]/Tripleo::Fencing/Pacemaker::Stonith::Level[stonith-1-525400e05b23]/Pcmk_stonith_level[stonith-level-1-$(/usr/sbin/crm_node -n)-stonith-fence_ipmilan-525400e05b23]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200305-60559-tn9ici create failed: Error: Stonith resource(s) 'stonith-fence_ipmilan-525400e05b23' do not exist, use --force to override"

    Nowhere in the code do we mandate that the single stonith resources need
    to be created *before* the stonith levels which make use of them.
    Let's add a constraint via collectors so we enforce this ordering.

    Tested on the environment that was not working and got a correctly
    deployed IHA overcloud.

    Change-Id: I78cb6ae21366a429b65a8357b3b267a485484a42
    Closes-Bug: #1866214

Changed in tripleo:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/711813

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/train)

Reviewed: https://review.opendev.org/711813
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=dd44d1552bb262adc7074d3567e034f83ec226dd
Submitter: Zuul
Branch: stable/train

commit dd44d1552bb262adc7074d3567e034f83ec226dd
Author: Michele Baldessari <email address hidden>
Date: Thu Mar 5 19:03:00 2020 +0100

    Make sure we create stonith resources before stonith levels

    Currently stonith levels kind of work out of pure luck. If puppet
    decides to reorder resources stonith levels can fail with:

                "Error: /Stage[main]/Tripleo::Fencing/Pacemaker::Stonith::Level[stonith-1-525400e05b23]/Pcmk_stonith_level[stonith-level-1-$(/usr/sbin/crm_node -n)-stonith-fence_ipmilan-525400e05b23]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200305-60559-tn9ici create failed: Error: Stonith resource(s) 'stonith-fence_ipmilan-525400e05b23' do not exist, use --force to override"

    Nowhere in the code do we mandate that the single stonith resources need
    to be created *before* the stonith levels which make use of them.
    Let's add a constraint via collectors so we enforce this ordering.

    Tested on the environment that was not working and got a correctly
    deployed IHA overcloud.

    Change-Id: I78cb6ae21366a429b65a8357b3b267a485484a42
    Closes-Bug: #1866214
    (cherry picked from commit b706f5365a84712eea8a133a9b8029878dcec2d2)

tags: added: in-stable-train
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/712568

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/stein)

Reviewed: https://review.opendev.org/712568
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=cbf11f05d461c9519a266f4cda91f01ad585d5c7
Submitter: Zuul
Branch: stable/stein

commit cbf11f05d461c9519a266f4cda91f01ad585d5c7
Author: Michele Baldessari <email address hidden>
Date: Thu Mar 5 19:03:00 2020 +0100

    Make sure we create stonith resources before stonith levels

    Currently stonith levels kind of work out of pure luck. If puppet
    decides to reorder resources stonith levels can fail with:

                "Error: /Stage[main]/Tripleo::Fencing/Pacemaker::Stonith::Level[stonith-1-525400e05b23]/Pcmk_stonith_level[stonith-level-1-$(/usr/sbin/crm_node -n)-stonith-fence_ipmilan-525400e05b23]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200305-60559-tn9ici create failed: Error: Stonith resource(s) 'stonith-fence_ipmilan-525400e05b23' do not exist, use --force to override"

    Nowhere in the code do we mandate that the single stonith resources need
    to be created *before* the stonith levels which make use of them.
    Let's add a constraint via collectors so we enforce this ordering.

    Tested on the environment that was not working and got a correctly
    deployed IHA overcloud.

    Change-Id: I78cb6ae21366a429b65a8357b3b267a485484a42
    Closes-Bug: #1866214
    (cherry picked from commit b706f5365a84712eea8a133a9b8029878dcec2d2)
    (cherry picked from commit dd44d1552bb262adc7074d3567e034f83ec226dd)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/712841

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/rocky)

Reviewed: https://review.opendev.org/712841
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=5de64973b083904ad48198d957df403f76928f1f
Submitter: Zuul
Branch: stable/rocky

commit 5de64973b083904ad48198d957df403f76928f1f
Author: Michele Baldessari <email address hidden>
Date: Thu Mar 5 19:03:00 2020 +0100

    Make sure we create stonith resources before stonith levels

    Currently stonith levels kind of work out of pure luck. If puppet
    decides to reorder resources stonith levels can fail with:

                "Error: /Stage[main]/Tripleo::Fencing/Pacemaker::Stonith::Level[stonith-1-525400e05b23]/Pcmk_stonith_level[stonith-level-1-$(/usr/sbin/crm_node -n)-stonith-fence_ipmilan-525400e05b23]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200305-60559-tn9ici create failed: Error: Stonith resource(s) 'stonith-fence_ipmilan-525400e05b23' do not exist, use --force to override"

    Nowhere in the code do we mandate that the single stonith resources need
    to be created *before* the stonith levels which make use of them.
    Let's add a constraint via collectors so we enforce this ordering.

    Tested on the environment that was not working and got a correctly
    deployed IHA overcloud.

    Change-Id: I78cb6ae21366a429b65a8357b3b267a485484a42
    Closes-Bug: #1866214
    (cherry picked from commit b706f5365a84712eea8a133a9b8029878dcec2d2)
    (cherry picked from commit dd44d1552bb262adc7074d3567e034f83ec226dd)
    (cherry picked from commit cbf11f05d461c9519a266f4cda91f01ad585d5c7)

tags: added: in-stable-rocky
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to puppet-tripleo (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/713037

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to puppet-tripleo (stable/queens)

Reviewed: https://review.opendev.org/713037
Committed: https://git.openstack.org/cgit/openstack/puppet-tripleo/commit/?id=8a899ae36b42798ac09773835b869abf7536c459
Submitter: Zuul
Branch: stable/queens

commit 8a899ae36b42798ac09773835b869abf7536c459
Author: Michele Baldessari <email address hidden>
Date: Thu Mar 5 19:03:00 2020 +0100

    Make sure we create stonith resources before stonith levels

    Currently stonith levels kind of work out of pure luck. If puppet
    decides to reorder resources stonith levels can fail with:

                "Error: /Stage[main]/Tripleo::Fencing/Pacemaker::Stonith::Level[stonith-1-525400e05b23]/Pcmk_stonith_level[stonith-level-1-$(/usr/sbin/crm_node -n)-stonith-fence_ipmilan-525400e05b23]/ensure: change from absent to present failed: pcs -f /var/lib/pacemaker/cib/puppet-cib-backup20200305-60559-tn9ici create failed: Error: Stonith resource(s) 'stonith-fence_ipmilan-525400e05b23' do not exist, use --force to override"

    Nowhere in the code do we mandate that the single stonith resources need
    to be created *before* the stonith levels which make use of them.
    Let's add a constraint via collectors so we enforce this ordering.

    Tested on the environment that was not working and got a correctly
    deployed IHA overcloud.

    Change-Id: I78cb6ae21366a429b65a8357b3b267a485484a42
    Closes-Bug: #1866214
    (cherry picked from commit b706f5365a84712eea8a133a9b8029878dcec2d2)
    (cherry picked from commit dd44d1552bb262adc7074d3567e034f83ec226dd)
    (cherry picked from commit cbf11f05d461c9519a266f4cda91f01ad585d5c7)
    (cherry picked from commit 5de64973b083904ad48198d957df403f76928f1f)

tags: added: in-stable-queens
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 12.2.0

This issue was fixed in the openstack/puppet-tripleo 12.2.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo 11.5.0

This issue was fixed in the openstack/puppet-tripleo 11.5.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo rocky-eol

This issue was fixed in the openstack/puppet-tripleo rocky-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo queens-eol

This issue was fixed in the openstack/puppet-tripleo queens-eol release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/puppet-tripleo stein-eol

This issue was fixed in the openstack/puppet-tripleo stein-eol release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.