After subcloud upgrade complete, password was not changed

[Heitor Matsui]

Brief Description
-----------------
NOTE: X and Y on the bug description was to hide the real passwords.

After DC sub cloud upgrade linux sysadmin user password was changed to X. During sub cloud upgrade password was Y and it was not reverted back to the original password X after upgrade is completed.

Severity
--------
Critical (password Y is predictable)

Steps to Reproduce
------------------
1. Install DC system with stx6 and the latest patches
2. Upgrade the Central cloud
3. Follow upgrade procedure for subcloud and complete upgrade and verify the login

Expected Behavior
------------------
Able to login as sysadmin user in subcloud using password X.

Actual Behavior
----------------
After upgrade Subcloud linux password changed to Y.

Reproducibility
---------------
Intermittent

System Configuration
--------------------
DC (subcloud)

Branch/Pull Time/Commit
-----------------------
master

Last Pass
---------
N/A

Timestamp/Logs
--------------
State: upgrading simplex, Subcloud: subcloud2, Details: get_subcloud_upgrade_data_install {'bmc_password': <redacted>, 'bootstrap-address': <redacted>, 'name': u'subcloud2', 'software_version': '22.06', 'wait_for_timeout': 3600, 'bmc_address': <redacted>, 'image': '/opt/dc-vault/loads/22.06/bootimage.iso', 'boot_device': u'/dev/disk/by-path/pci-0000:03:00.0-nvme-1', 'ansible_ssh_pass': Y, 'rootfs_device': u'/dev/disk/by-path/pci-0000:03:00.0-nvme-1', 'install_type': 4, 'bootstrap_address': <redacted>, 'no_check_certificate': True, 'admin_password': X, 'ansible_become_pass': Y, 'bootstrap_address_prefix': 64, 'bootstrap_interface': u'eno1', 'bmc_username': u'sysadmin'}

Test Activity
-------------
Feature Testing

Workaround
----------
Verify the password on all subclouds after upgrade and change accordingly

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/config/+/880784
Committed: https://opendev.org/starlingx/config/commit/b887d9961ef08404fb60fa91c09cc59a4ff65dc9
Submitter: "Zuul (22348)"
Branch: master

commit b887d9961ef08404fb60fa91c09cc59a4ff65dc9
Author: Heitor Matsui <email address hidden>
Date: Tue Apr 18 19:07:15 2023 -0300

    Audit local sysadmin user password periodically

    The platform as designed currently does not update the user
    configuration as an atomic operation, as it first update the
    database on [1] and then later run a manifest to configure
    the hosts on [2]. If the database is updated but the manifest
    apply fails, the hosts may remain with outdated user data
    locally.

    This commit adds an agent audit to periodically check if local
    user data is consistent with the database, and if it is not,
    then it calls the platform::users::runtime manifest to attempt
    to reconfigure the user correctly on the host.

    To enable this audit, this commit also adds a new RPC call to
    allow the agent to receive user config from the database, and
    because of this new RPC call, when upgrading from older loads,
    there will be an AttributeError exception in the active controller
    logs that is harmless and should be cleared when both controllers
    are upgraded, and the RPC call update_user_config is changed to
    allow running the runtime manifest only for a group of hosts.

    [1] https://opendev.org/starlingx/config/src/commit/2afa67e8730b62465b02041865934b4d42ded3f7/sysinv/sysinv/sysinv/sysinv/api/controllers/v1/user.py#L240
    [2] https://opendev.org/starlingx/config/src/commit/2afa67e8730b62465b02041865934b4d42ded3f7/sysinv/sysinv/sysinv/sysinv/conductor/manager.py#L8147

    Test Plan:
    PASS: set log to debug and verify audit executing on the log
    PASS: update OS user sysadmin's password on active controller to
          trigger the sysinv host update call, force manifest apply
          failure on a host and verify that agent audit on the host
          detects the difference and attempts to reapply the manifest,
          then verify that the password is updated on the host

    Regression:
    PASS: update OS user sysadmin's password on active controller,
          verify manifest being reapplied on all hosts
    PASS: upgrade AIO-SX stx6 -> stx8
    PASS: upgrade AIO-DX stx6 -> stx8
    PASS: upgrade DC central cloud + orchestrated subcloud stx6 -> stx8

    Closes-bug: 2017039

    Change-Id: Iaa33dac08a8d246be366ee93ab0507fecddaeb4e
    Signed-off-by: Heitor Matsui <email address hidden>