mtcClient seen to sometimes start flooding the mtcAgent with Host Services Result messages

[Eric MacDonald]

Brief Description
-----------------
A recent update [1] that enabled logging receipt of Host Services Result messages revealed that the mtcClient sometimes gets into a state where it starts flooding the mtcAgent with Host Services Result messages.

[1] https://opendev.org/starlingx/metal/commit/fd665193399d85265c6ec7e74b9558f597dbb270

Severity
--------
Major: Unnecessary messaging that leads to mtcAgent log flooding

Steps to Reproduce
------------------
Lock/Unlock node

Expected Behavior
-----------------
mtcClient does not flood mtcAgent with said messages

Actual Behavior

mtcAgent gets flooded with said messages and floods its logfile with message received logs.

Reproducibility
---------------
Intermittent, likely due to the race condition that lead to the need for the mentioned update.

System Configuration
--------------------
Multi-node systems

Load info (eg: 2022-03-10_20-00-07)
---------
Suspect introduced by this update delivered on June 4, 2024

https://opendev.org/starlingx/metal/commit/1335bc484df331771e995ae822df3af84cc5739d

Last Pass
---------
Silent behavior prior to addition of update mentioned in description

Timestamp/Logs
--------------
The mtcAgent.log shows this log every frequently. Issue was traced back to the mtcClient sending this message that frequently

2024-07-24T16:07:59.365 [28489.00199] controller-0 mtcAgent msg mtcCtrlMsg.cpp ( 300) mtc_service_inbox : Info : controller-1 'host services result' ACK (rc:0) (Mgmnt)
2024-07-24T16:07:59.365 [28489.00200] controller-0 mtcAgent msg mtcCtrlMsg.cpp ( 300) mtc_service_inbox : Info : controller-1 'host services result' ACK (rc:0) (Mgmnt)
2024-07-24T16:07:59.365 [28489.00201] controller-0 mtcAgent msg mtcCtrlMsg.cpp ( 300) mtc_service_inbox : Info : controller-1 'host services result' ACK (rc:0) (Mgmnt)
2024-07-24T16:07:59.365 [28489.00202] controller-0 mtcAgent msg mtcCtrlMsg.cpp ( 300) mtc_service_inbox : Info : controller-1 'host services result' ACK (rc:0) (Mgmnt)

This mtcAgent log shows the excessive messaging and rate

2024-07-24T16:33:54.323 [28489.32944] controller-0 mtcAgent mtc mtcNodeCtrl.cpp (1718) daemon_service_run : Info : 2357 messages processed ; rate: 39.3 msgs/sec] [2335:13:0:9:0:0]

Alarms
------
None

Test Activity
-------------
Developer Testing

Workaround
----------
sudo pmon-restart mtcClient on the node sending messages