ipmitool "timing" flags are not working as expected causing failure to manage power of baremetal nodes

[Drew Freiberger]

In a focal-ussuri cloud environment where there is some amount of packet loss between the ironic-conductor and the BMC network, I'm experiencing random timeout issues with ipmitool failures.

The root issue I'm having is that using:

ipmitool -R 12 -N 5 <command>

is resulting in ipmitool hanging for 60 seconds (12 commands are sent even though the session is never properly started) and then timing out within the ironic-conductor application, causing "clean failed" state when transitioning a node from 'manage' to 'provide' status.

Ultimately, it appears that ussuri runs this bit of code that determines that ipmitool accepts -R and -N flags and instead of performing retries of ipmitool within the ironic code, it relies on ipmitool to perform all of the retries.

https://opendev.org/openstack/ironic/src/branch/stable/ussuri/ironic/drivers/modules/ipmitool.py#L538-L546

This has been addressed in the mainline code by the addition of an operator configurable option 'use_ipmitool_retries' to let ipmitool perform retries via -R flag, or fall back to letting ironic execute ipmitool multiple separate times.

https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/ipmitool.py#L494

In my environment, I require to re-run ipmitool multiple separate times to avoid failure.

Can we please backport this functionality into focal-ussuri?
https://opendev.org/openstack/ironic/commit/1de3db3b16f3e0475e506e540ca5d5ed6edb4cbf

Also, please expose charm configuration to allow operator to set "[ipmi] use_ipmitool_retries" = False.

[Drew Freiberger (afreiberger)]

FYI, the commit with the option is available in Victoria+

[Seyeong Kim (seyeongkim)]

@afreiberger

Hello, Do you have test env?

I've created PPA for testing but I don't have test env yet.

If you have, Could you please test it?I put the commit you mentioned and 2 more related commits.

https://launchpad.net/~seyeongkim/+archive/ubuntu/lp1943765-2

[Hemanth Nakkina (hemanth-n)]

I have verified ironic-conductor ipmitool commands behaviour with the above PPA in #2 (On focal ussuri)

With configuration use_ipmitool_retries = False, ironic-conductor runs below command until 60 seconds timeout expiry.
Command: ipmitool -I lanplus -H 10.5.0.5:9999 -L ADMINISTRATOR -U test -R 1 -N 5 -f /tmp/tmpmt5292he power status

OpenStack commands used for testing:
/snap/bin/openstack baremetal node create --driver ipmi --driver-info ipmi_address=10.5.0.5:9999 --driver-info ipmi_username=test --driver-info ipmi_password=test
/snap/bin/openstack baremetal node list
/snap/bin/openstack baremetal node power on <node id>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/810610

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-conductor/+/810610
Committed: https://opendev.org/openstack/charm-ironic-conductor/commit/73a5b90d4026b5acf2cefe1f1057d078c8e923e4
Submitter: "Zuul (22348)"
Branch: master

commit 73a5b90d4026b5acf2cefe1f1057d078c8e923e4
Author: Hemanth Nakkina <email address hidden>
Date: Thu Sep 23 16:49:14 2021 +0530

    Add support for new option use-ipmitool-retries

    Add new configuration option use-ipmitool-retries to the charm.

    Closes-Bug: #1943765
    Change-Id: I2d11198d1955f3b96d27163683ac0947639d2f74