Mtce is not notified of BMC password change

[Eric MacDonald]

Brief Description
-----------------
Maintenance is not sent any notification when there is a BMC password only provisioning change.

Severity
--------
Minor:

Rare but inconvenient.
Hard to debug since password information is never logged.
Issue just shows up as a system call failure.
Customer can de-provision and re-provision as work around

Steps to Reproduce
------------------
system host-update controller-0 bm_type=bmc bm_password=root bm_username=root bm_ip=128.224.149.132
sleep 30
system host-update controller-0 bm_type=bmc bm_password=root1 bm_username=root bm_ip=128.224.149.132

Expected Behavior
------------------
Maintenance should get notification of BMC provisioning change.

Actual Behavior
----------------
Maintenance does not get notification of BMC provisioning change.

Reproducibility
---------------
Reproducible: 100%

System Configuration
--------------------
Systems with provisioned BMCs

Branch/Pull Time/Commit
-----------------------
All

Last Pass
---------
Never - has always been broken
UT and PV test escape

Timestamp/Logs
--------------
Tail sysinv.log and mtcAgent.log file at provisioning time.

Test Activity
-------------
Unit Test

[Eric MacDonald (rocksolidmtce)]

The BMC password is not included in the data that comes from sysinv.

Solution will require some means of notifying mtce that the password has changed.

Another alternative would be for maintenance to automatically re-fetch the BMC password when BMC access requests fail.

[Eric MacDonald (rocksolidmtce)]

Implementing change that re-fetches the BMC password on failed BMC access requests.

[Ghada Khalil (gkhalil)]

Low priority / not gating - but would be nice to fix

[Eric MacDonald (rocksolidmtce)]

I found a more serious issue with this.
It turns out that barbican is not updated with a new password if only the bm_password is changed.

So if the customer provisions the BMC and accidentally fat fingers the password, he/she cannot simply command line up, change the password to the correct value and re-execute.

Instead the username or password would also have to be changes and then changed back after the good password is stored.

The other alternative is to deprovision completely and then reprovision.

I think this is higher than a Low priority.

I will deliver the change to maintenance that re-fetches the password on a reconnect but that still won't fix the password only change behavior.

[Eric MacDonald (rocksolidmtce)]

This issue number will be used to deliver the update that re-fetches the BMC password on a connection error.

Will raise a new bug for the inability to perform a bmc password only change.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/686435

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/686435
Committed: https://git.openstack.org/cgit/starlingx/metal/commit/?id=4c541f50d4073278d810c04665f14a9060ce7a2c
Submitter: Zuul
Branch: master

commit 4c541f50d4073278d810c04665f14a9060ce7a2c
Author: Eric MacDonald <email address hidden>
Date: Thu Oct 3 11:56:42 2019 -0400

    Maintenance Redfish support useability enhancements.

    This update is a result of changes made during a suite of
    end-to-end provisioning, reprovisioning and deprovisioning
    customer exterience testing of the maintenance RedFish support
    feature.

    1. Force reconnection and password fetch on provisioning changes
    2. Force reconnection and password fetch on persistent connection failures
    3. Fix redfish protocol learning (string compare) in hardware monitor
    4. Improve logging for some typical error paths.

    Test Plan:

    PASS: Verify handling of reprovisioning BMC between hosts that support
                 different protocols.
    PASS: Verify handling of reprovisioning ip address to host that leads to a
                 different protocol select.
    PASS: Verify manual relearn handling to recover from errors that result from
                 the above case.
    PASS: Verify host BMC deprovisioning handling and cleanup.
    PASS: Verify sensor monitoring.
    PASS: Verify hwmond sticks with a selected protocol once a sensor model
                 has been created using that protocol.
    PASS: Verify handling of BMC reprovision - ip address change only
    PASS: Verify handling of BMC reprovision - username change only
    FAIL: Verify handling of BMC reprovision - password change only
                 https://bugs.launchpad.net/starlingx/+bug/1846418

    Change-Id: I4bf52a5dc3c97d7794ff623c881dff7886234e79
    Closes-Bug: #1846212
    Story: 2005861
    Task: 36606
    Signed-off-by: Eric MacDonald <email address hidden>