Mtce reinstall fails for some servers that are powered off

[Eric MacDonald]

The BMC of some servers, like HPEs, silently reject 'set next boot device' and possibly other board management commands while it is in what is called 'POST Mode' ; aka executing BIOS. This is the time immediately following 'power-on' or 'reset' and continues until the server executes Linux (from disk or iso image).

If the Maintenance Reinstall Handler detects that a server's power is off it will issue a 'power-on' board management command which immediately puts the server into 'POST Mode' (BIOS) where it silently rejects the next board management command to 'set next boot device to pxe' ; which is followed by 'reset'.

Since the 'set next boot device' command did not occur (was silently rejected) the server will boot a valid image on disk if it exists thereby failing the intended reinstall operation.

A similar issue was detected during the development of the 'rvmc' (Redfish Virtual Media Controller) container for subcloud install.

Issue was corrected by switching to an algorithm that always powered the server off first, followed by other board management commands then followed by a power on.

The Maintenance Reinstall Handler will also need to make a similar change.

New Algorithm:

Step 1. Power Off Host
Step 2. Wait for Shutdown
Step 3. Verify Power Off
Step 4. Set next boot device to pxe
Step 5. Power on Host

With this algorithm the host will always Network Boot regardless of the initial power state of the host.

Severity:
---------
Major (has work around)

Work Around:
------------
Power On server, wait for BIOS to complete , issue Reinstall command.

Steps to Reproduce:
--------------------
Power off server in advance of issuing Maintenance Host Reinstall command/operation.

Expected Behavior:
------------------
Host boots from network even if valid image on disk.

Actual Behavior
----------------
Host boots from local disk instead of from network.

Reproducibility:
----------------
100% of time if Host to be reinstalled has a valid image on disk and is powered off in advance of the reinstall operation.

System Configuration
--------------------
Any system with HPE Servers.

Branch/Pull Time/Commit
-----------------------
Lab/Server Specific.
Since the development of the deployment manager which added forced network boot to the Maintenance Reinstall Handler.

Last Pass
---------
Newly observed behavior on the HP380 lab using the new auto install with advanced power off algorithm.

Timestamp/Logs
--------------
Not Required.

Test Activity
-------------
Auto Installation with advanced host power-off changes.

[Ghada Khalil (gkhalil)]

stx.4.0 / medium priority - there is a workaround

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/706895

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/706895
Committed: https://git.openstack.org/cgit/starlingx/metal/commit/?id=da7b2e94f136fa9faf7ff691f175b2ca9b1605b1
Submitter: Zuul
Branch: master

commit da7b2e94f136fa9faf7ff691f175b2ca9b1605b1
Author: Eric MacDonald <email address hidden>
Date: Mon Feb 10 10:15:56 2020 -0500

    Modify Mtce Reinstall FSM to first power-off BMC provisioned hosts

    This update only applies to servers that support and are provisioned
    for Board Management Control (BMC).

    The BMC of some servers silently reject the 'set next boot device',
    a command while it is executing BIOS.

    The current reinstall algorithm when the BMC is provisioned starts by
    detecting the power state of the target server. If the power is off
    it will 'first power it on' and then proceed to 'set next boot device'
    to pxe followed by a reset. For the initial power off state case, the
    timing of these operations is such that the server is in BIOS when the
    'set next boot device' command is issued.

    This update modifies the host reinstall algorithm to first power-off
    a server followed by setting the next boot device while the server is
    confirmed to be powered off, then powered on. This ensures the server
    gets and handles the set next boot device command operation properly.

    This update also fixes a race condition between the bmc_handler and
    power_handler by moving the final power state update in the power
    handler to the power done phase.

    Test Plan:

    Verify all new reinstall failure path handling via fault insertion testing
    Verify reinstall of powered off host
    Verify reinstall of powered on host
    Verify reinstall of Wildcat server with ipmi
    Verify reinstall of Supermicro server with ipmi and redfish
    Verify reinstall of Ironpass server with ipmi
    Verify reinstall of WolfPass server with redfish and ipmi
    Verify reinstall of Dell server with ipmi

    Over 30 reinstalls were performed across all server types, with initial
    power on and off using both ipmi and redfish (where supported).

    Change-Id: Iefb17e9aa76c45f2ceadf83f23b1231ae82f000f
    Closes-Bug: 1862065
    Signed-off-by: Eric MacDonald <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: f/centos8
Review: https://review.opendev.org/716148

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/716148
Committed: https://git.openstack.org/cgit/starlingx/metal/commit/?id=5d064c51333cc8acebbb5c837b6df2cf767e5d3f
Submitter: Zuul
Branch: f/centos8

commit 041049063e81c17d9c513a10dca9f073139a979b
Author: Don Penney <email address hidden>
Date: Wed Feb 5 14:36:43 2020 -0500

    Update pxeboot kickstart to allow for hybrid install

    The generated pxeboot kickstart templates that are included in the
    installation ISO will be used by the gen-bootloader-iso.sh in setting
    up a hybrid ISO/network installation, where the kernel and initrd are
    loaded from ISO, but the rootfs, kickstart, and software repositories
    are accessed via http/https. In order to support this installation,
    the pxeboot kickstart is updated to allow for the case where the
    BOOTIF cmdline parameter is not set (ie. node booted from ISO rather
    than network).

    This update also adds a check to the pxeboot kickstart to mount the
    installation media if in this scenario. In the case where a controller
    has been booted from an ISO with installation media accessed from the
    network, dracut does not mount the ISO. This results in the 'reboot
    --eject' in the kickstart being ignored, and when the newly installed
    node is rebooted, it may just boot the ISO again. By mounting the
    media in this case, Anaconda will eject it as expected.

    Story: 2006980
    Task: 38466

    Change-Id: I9330a2d50b8966a2b3f2b59cabec20f084372d8c
    Signed-off-by: Don Penney <email address hidden>

commit da7b2e94f136fa9faf7ff691f175b2ca9b1605b1
Author: Eric MacDonald <email address hidden>
Date: Mon Feb 10 10:15:56 2020 -0500

    Modify Mtce Reinstall FSM to first power-off BMC provisioned hosts

    This update only applies to servers that support and are provisioned
    for Board Management Control (BMC).

    The BMC of some servers silently reject the 'set next boot device',
    a command while it is executing BIOS.

    The current reinstall algorithm when the BMC is provisioned starts by
    detecting the power state of the target server. If the power is off
    it will 'first power it on' and then proceed to 'set next boot device'
    to pxe followed by a reset. For the initial power off state case, the
    timing of these operations is such that the server is in BIOS when the
    'set next boot device' command is issued.

    This update modifies the host reinstall algorithm to first power-off
    a server followed by setting the next boot device while the server is
    confirmed to be powered off, then powered on. This ensures the server
    gets and handles the set next boot device command operation properly.

    This update also fixes a race condition between the bmc_handler and
    power_handler by moving the final power state update in the power
    handler to the power done phase.

    Test Plan:

    Verify all new reinstall failure path handling via fault insertion testing
    Verify reinstall of powered off host
    Verify reinstall of powered on host
    Verify reinstall of Wildcat server with ipmi
    Verify reinstall of Supermicro ...