drbdadm stacktraces in sysinv.log

[Al Bailey]

Brief Description
-----------------
A stacktrace occurs regularly in sysinv.log after an AIO-SX is unlocked

Severity
--------
Minor

Steps to Reproduce
------------------
Build and boot a system after March 27 2023, unlock AIO-SX and wait a minute

Expected Behavior
------------------
sysinv.log should not show FileNotFound exceptions for drbdadm

Actual Behavior
----------------
lots of them

Reproducibility
---------------
100%

System Configuration
--------------------
AIO-SX

Branch/Pull Time/Commit
-----------------------
March 27, 2023

Last Pass
---------
March 26, 2023

Timestamp/Logs
--------------

sysinv 2023-03-27 20:21:00.744 64135 INFO sysinv.agent.manager [-] _apply_runtime_manifest with hieradata_path = '/opt/platform/puppet/22.12/hieradata'
sysinv 2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task [-] Error during ConductorManager._controller_config_active_apply: FileNotFoundError: [Errno 2] No such file or directory: 'drbdadm'
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task Traceback (most recent call last):
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task task(self, context)
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/conductor/manager.py", line 5928, in _controller_config_active_apply
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task all_fs_resized = self._resize_filesystems_update_db(context, standby_host)
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/conductor/manager.py", line 5959, in _resize_filesystems_update_db
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task all_fs_resized, drbd_fs_resized = self._config_resize_filesystems(context, standby_host)
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/conductor/manager.py", line 10754, in _config_resize_filesystems
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task stdout, __ = cutils.execute(*cmd, attempts=retry_attempts, run_as_root=True)
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/sysinv/common/utils.py", line 249, in execute
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task obj = subprocess.Popen(cmd,
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task File "/usr/lib/python3/dist-packages/eventlet/green/subprocess.py", line 58, in __init__
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task self._execute_child(args, executable, preexec_fn, close_fds,
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task File "/usr/lib/python3.9/subprocess.py", line 1823, in _execute_child
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task raise child_exception_type(errno_num, err_msg, err_filename)
2023-03-27 20:21:00.757 69763 ERROR oslo_service.periodic_task FileNotFoundError: [Errno 2] No such file or directory: 'drbdadm'

Test Activity
-------------
Developer Testing

Workaround
----------
Revert https://review.opendev.org/c/starlingx/config/+/869109

The problem is that that change introduces a default empty dictionary for the environment variable in subprocess.Popen

drbdadm is located under /usr/sbin, and that PATH is not in the env, it cannot be found
This problem likely impacts many methods that use that sysinv.common.utils "execute" method

[Ghada Khalil (gkhalil)]

The code that introduced this issue has been reverted. Review: https://review.opendev.org/c/starlingx/config/+/878844

Based on the revert, this issue can be marked as Fix Released since it's been addressed. The original code ( review: https://review.opendev.org/c/starlingx/config/+/869109 ) will need to be re-worked to avoid this issue on resubmission.