host unlock failure on standby controller with stx. 6.0 load during upgrade

[Luis Eduardo Angelini Marquitti]

Brief Description
-----------------
There is a test to lock and unlock the standby controller before the upgrade. Host unlock and force host unlock were failing on controller-0 before the upgrade. This was observed during the stx. 6.0 to stx. 7.0 upgrade.

Severity
--------
Major

Steps to Reproduce
------------------
1. Install AIO-DX or DC system with stx. 6.0.
2. Follow the upgrade procedure to upgrade up to controller-1 and swact controller-1 to keep controller-1 active.
3. Lock controller-0 and unlock controller-0. Which is failed. Tried to force unlock controller-0 also failed. As per description.

Expected Behavior
------------------
Able to lock and unlock. Force lock and unlock should be able to do without error.

Actual Behavior
----------------
As per the description failure on unlock.

Reproducibility
---------------
Reproducible

System Configuration
--------------------
AIO-DX or DC stx. 6.0 to stx. 7.0 upgrade.

Branch/Pull Time/Commit
-----------------------
stx. 6.0

Last Pass
---------
This was passed stx. 6.0 release test

Timestamp/Logs
--------------
system host-unlock controller-0 -f [Errno 2] No such file or directory iserver-advertise-address abcd:205::2 --cri-socket /var/run/containerd/containerd.sock', 'platform::kubernetes::params::version': 'v1.21.8'}

sysinv 2022-06-29 14:06:00.864 293942 INFO sysinv.puppet.puppet [-] Updating hiera for host: controller-0 with config_uuid: 8f9dd220-4e42-47c3-9687-2e78ad8cfa12 target_load: 21.12 config: {'platform::kubernetes::params::join_cmd': 'kubeadm join [abcd:205::1]:6443 --token ii1blv.issc6kxk5yj6m83r --discovery-token-ca-cert-hash sha256:4306ab05903dbfba5e56762e38814df455f374a14f48c8a56633945ea45c42ca --control-plane --certificate-key ffe79803e04c312519de34fc23da2e5bc8f4c4fbd8337f0e6516e1ad8f5b2d0a --apiserver-advertise-address abcd:205::2 --cri-socket /var/run/containerd/containerd.sock', 'platform::kubernetes::params::version': 'v1.21.8', 'platform::config::params::config_uuid': '8f9dd220-4e42-47c3-9687-2e78ad8cfa12'}
sysinv 2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp [-] Exception during message handling: OSError: [Errno 2] No such file or directory
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp Traceback (most recent call last):
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 436, in _process_data
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     **args)
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 2103, in configure_ihost
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     self._configure_controller_host(context, host)
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1562, in _configure_controller_host
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     self._update_pxe_config(host)
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1219, in _update_pxe_config
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     stderr=fnull)
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/subprocess.py", line 537, in check_call
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     retcode = call(*popenargs, **kwargs)
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/subprocess.py", line 524, in call
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     return Popen(*popenargs, **kwargs).wait()
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 58, in _{_}init{_}_
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     subprocess_orig.Popen._{_}init{_}_(self, args, 0, *argss, **kwds)
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/subprocess.py", line 711, in _{_}init{_}_
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     errread, errwrite)
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp   File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp     raise child_exception
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory
2022-06-29 14:06:01.664 293942 ERROR sysinv.openstack.common.rpc.amqp
sysinv 2022-06-29 14:06:01.666 293942 ERROR sysinv.openstack.common.rpc.common [-] Returning exception [Errno 2] No such file or directory to caller: OSError: [Errno 2] No such file or directory
sysinv 2022-06-29 14:06:01.666 293942 ERROR sysinv.openstack.common.rpc.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/amqp.py", line 436, in {_}process_data\n    **args)\n', '  File "/usr/lib64/python2.7/site-packages/sysinv/openstack/common/rpc/dispatcher.py", line 172, in dispatch\n    result = getattr(proxyobj, method)(ctxt, **kwargs)\n', '  File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 2103, in configure_ihost\n    self._configure_controller_host(context, host)\n', '  File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1562, in _configure_controller_host\n    self._update_pxe_config(host)\n', '  File "/usr/lib64/python2.7/site-packages/sysinv/conductor/manager.py", line 1219, in _update_pxe_config\n    stderr=fnull)\n', '  File "/usr/lib64/python2.7/subprocess.py", line 537, in check_call\n    retcode = call(*popenargs, **kwargs)\n', '  File "/usr/lib64/python2.7/subprocess.py", line 524, in call\n    return Popen(*popenargs, **kwargs).wait()\n', '  File "/usr/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 58, in __init{_}{_}\n    subprocess_orig.Popen.{_}{_}init{_}{_}(self, args, 0, *argss, **kwds)\n', '  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init{_}_\n    errread, errwrite)\n', '  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child\n    raise child_exception\n', 'OSError: [Errno 2] No such file or directory\n']: OSError: [Errno 2] No such file or directory

Test Activity
-------------
Upgrade Regression testing

Workaround
----------
-

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/854789

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Luis Eduardo Angelini Marquitti <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/854789
Reason: Already fixed by https://review.opendev.org/c/starlingx/config/+/829943