overcloud generate fencing command cannot be run before the actual deployment

[Michele Baldessari]

Currently 'openstack overcloud generate fencing instackenv.json --output /tmp/fence.yaml' needs to run after the actual deployment and then you need to run another deploy command with the added /tmp/fence.yaml environment file. While this used to be necessary in environments without IPMI, this
is not strictly necessary since we've moved to VBMC (and in real life IPMI was the reality anyway).

The reason we typically could not generate this fencing information before hand was that before the deployment we did not have the correspondence between ironic and the actual nova node name before deployment.

The way this can be solved is by simply using the mac address and then let puppet::tripleo match its own node and stonith info by looking up its mac addresses during deploy and matching them to its hostname.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/511423
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=19f3e7a325696cf33a8d85a0d4ccc8358f750f1d
Submitter: Zuul
Branch: master

commit 19f3e7a325696cf33a8d85a0d4ccc8358f750f1d
Author: Chris Jones <email address hidden>
Date: Thu Oct 12 09:35:59 2017 +0100

    Allow fencing config generation before deployment.

    With this commit we are able to generate the fencing configuration for
    IPMI hosts before deployment, and thus use it as part of the initial
    deployment run.

    The reason we can generate it before hand is the following:
    puppet::tripleo is capable of matching its own host via
    https://github.com/openstack/puppet-tripleo/blob/master/manifests/fencing.pp#L73
    and by doing that each node will create its own stonith device.
    It does so by looking at the macaddress<->IPMI table and if it detects
    it's own macaddress it will create the IPMI stonith device
    for its own hostname.

    Concurrent stonith resource creation from different nodes is possible
    in puppet-pacemaker since change I8be5d5d1a9894b0e2915459b10ea2feed703ba8e
    got merged.

    Tested as follows on a virtual vbmc environment:
    1. Before deployment:
    $ openstack overcloud generate fencing instackenv.json --output fence.yaml
    2. Deployed overcloud adding "fence.yaml"
    3. Verified stonith configuration:
    [root@controller-0 ~]# pcs status |grep stonith
     stonith-fence_ipmilan-525400c36fc9 (stonith:fence_ipmilan): Started controller-0
     stonith-fence_ipmilan-5254005c160d (stonith:fence_ipmilan): Started controller-2
     stonith-fence_ipmilan-525400ed3293 (stonith:fence_ipmilan): Started controller-1
    [root@controller-0 ~]# pcs property |grep stonith
     stonith-enabled: true
    4. Verified the fencing of a specific node:
    [root@controller-0 ~]# pcs stonith fence controller-1
    Node: controller-1 fenced

    <indeed controller-1 has been fenced>

    Same test run on BM:
    1. Before deployment:
    $ openstack overcloud generate fencing instackenv.json --output fence.yaml
    2. Deployed overcloud adding "fence.yaml"
    3. Verified stonith configuration:
    [root@controller-0 ~]# pcs status |grep stonith
     stonith-fence_ipmilan-1866da6126a0 (stonith:fence_ipmilan): Started overcloud-controller-0
     stonith-fence_ipmilan-1866da612109 (stonith:fence_ipmilan): Started overcloud-controller-1
     stonith-fence_ipmilan-1866da5faed8 (stonith:fence_ipmilan): Started overcloud-controller-2
     stonith-fence_ipmilan-1866da612373 (stonith:fence_ipmilan): Started overcloud-controller-2
     stonith-fence_ipmilan-1866da612295 (stonith:fence_ipmilan): Started overcloud-controller-0
    [root@overcloud-controller-0 ~]# pcs property |grep stonith
     stonith-enabled: true
    4. Verified the fencing of a specific node:
    [root@overcloud-controller-0 ~]# pcs stonith fence overcloud-controller-1 & ping overcloud-controller-1
    [1] 168504
    PING overcloud-controller-1.localdomain (172.17.0.14) 56(84) bytes of data.
    64 bytes from ove...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/566554

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/566554
Committed: https://git.openstack.org/cgit/openstack/tripleo-common/commit/?id=1d3aefbe2f0aac2828eba69ee9efc57a7b7bf385
Submitter: Zuul
Branch: stable/queens

commit 1d3aefbe2f0aac2828eba69ee9efc57a7b7bf385
Author: Chris Jones <email address hidden>
Date: Thu Oct 12 09:35:59 2017 +0100

    Allow fencing config generation before deployment.

    With this commit we are able to generate the fencing configuration for
    IPMI hosts before deployment, and thus use it as part of the initial
    deployment run.

    The reason we can generate it before hand is the following:
    puppet::tripleo is capable of matching its own host via
    https://github.com/openstack/puppet-tripleo/blob/master/manifests/fencing.pp#L73
    and by doing that each node will create its own stonith device.
    It does so by looking at the macaddress<->IPMI table and if it detects
    it's own macaddress it will create the IPMI stonith device
    for its own hostname.

    Concurrent stonith resource creation from different nodes is possible
    in puppet-pacemaker since change I8be5d5d1a9894b0e2915459b10ea2feed703ba8e
    got merged.

    Tested as follows on a virtual vbmc environment:
    1. Before deployment:
    $ openstack overcloud generate fencing instackenv.json --output fence.yaml
    2. Deployed overcloud adding "fence.yaml"
    3. Verified stonith configuration:
    [root@controller-0 ~]# pcs status |grep stonith
     stonith-fence_ipmilan-525400c36fc9 (stonith:fence_ipmilan): Started controller-0
     stonith-fence_ipmilan-5254005c160d (stonith:fence_ipmilan): Started controller-2
     stonith-fence_ipmilan-525400ed3293 (stonith:fence_ipmilan): Started controller-1
    [root@controller-0 ~]# pcs property |grep stonith
     stonith-enabled: true
    4. Verified the fencing of a specific node:
    [root@controller-0 ~]# pcs stonith fence controller-1
    Node: controller-1 fenced

    <indeed controller-1 has been fenced>

    Same test run on BM:
    1. Before deployment:
    $ openstack overcloud generate fencing instackenv.json --output fence.yaml
    2. Deployed overcloud adding "fence.yaml"
    3. Verified stonith configuration:
    [root@controller-0 ~]# pcs status |grep stonith
     stonith-fence_ipmilan-1866da6126a0 (stonith:fence_ipmilan): Started overcloud-controller-0
     stonith-fence_ipmilan-1866da612109 (stonith:fence_ipmilan): Started overcloud-controller-1
     stonith-fence_ipmilan-1866da5faed8 (stonith:fence_ipmilan): Started overcloud-controller-2
     stonith-fence_ipmilan-1866da612373 (stonith:fence_ipmilan): Started overcloud-controller-2
     stonith-fence_ipmilan-1866da612295 (stonith:fence_ipmilan): Started overcloud-controller-0
    [root@overcloud-controller-0 ~]# pcs property |grep stonith
     stonith-enabled: true
    4. Verified the fencing of a specific node:
    [root@overcloud-controller-0 ~]# pcs stonith fence overcloud-controller-1 & ping overcloud-controller-1
    [1] 168504
    PING overcloud-controller-1.localdomain (172.17.0.14) 56(84) bytes of data.
    64 bytes f...

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 8.6.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-common 9.1.0 release.