problems after changing expired password

[George Postolache]

Brief Description
-----------------
After changing the expired password system is not responsive anymore (unable to open new ssh connections to controller-1 or controller-0). On the previously opened ssh connection to the active controller commands are not working anymore. After restarting controller-1 via ipmitool controller-1 comes back on, controller-0 is also automatically restarted and the new password works on both of them.
After this an stx-openstack apply is automatically triggered and failed. Further app removes are also failed, but after a force lock/unlock of controller-0 i was able to remove/apply stx-openstack and get the sistem back online

Severity
--------
Provide the severity of the defect.
<Minor: System/Feature is usable with minor issue>

Steps to Reproduce
------------------
1. get curent date
2. disable ntp:
  sudo timedatectl set-ntp 0
3. Check pass expiration time
  chage -l sysadmin
4. Change date to a date after the expiration date
  sudo timedatectl set-time 2021-05-21
5. Use "sudo timedatectl set-status" to check the time is set and to get the prompt to change the expired password and change the password
6. Change the date back to the original date
  sudo timedatectl set-time <original date>
7. enable ntp
  sudo timedatectl set-ntp 1
8. try to lock/unlock standby controller and swact via horizon gui

Expected Behavior
------------------
locking/unlocking standby controller and swacting to it should be possible

Actual Behavior
----------------
commands are failing on controller-0 and new ssh connections with the controllers are not successful (you don't get the prompt)

Reproducibility
---------------
always

Branch/Pull Time/Commit
-----------------------
###
### StarlingX
### Built from master
###

OS="centos"
SW_VERSION="21.05"
BUILD_TARGET="Host Installer"
BUILD_TYPE="Formal"
BUILD_ID="20210401T221102Z"

JOB="STX_build_layer_flock_master_master"
<email address hidden>"
BUILD_NUMBER="444"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2021-04-01 22:11:02 +0000"

FLOCK_OS="centos"
FLOCK_JOB="STX_build_layer_flock_master_master"
<email address hidden>"
FLOCK_BUILD_NUMBER="444"
FLOCK_BUILD_HOST="starlingx_mirror"
FLOCK_BUILD_DATE="2021-04-01 22:11:02 +0000"

DISTRO_OS="centos"
DISTRO_JOB="STX_build_layer_distro_master_master"
<email address hidden>"
DISTRO_BUILD_NUMBER="454"
DISTRO_BUILD_HOST="starlingx_mirror"
DISTRO_BUILD_DATE="2021-04-01 20:07:32 +0000"

COMPILER_OS="centos"
COMPILER_JOB="STX_build_layer_compiler_master_master"
<email address hidden>"
COMPILER_BUILD_NUMBER="512"
COMPILER_BUILD_HOST="starlingx_mirror"
COMPILER_BUILD_DATE="2021-04-01 18:14:16 +0000"

Test Activity
-------------
Regression Testing

Workaround
----------
reboot controller-1 using impitool or press the button
wait for controller-1 to come back online and controller-0 to auto-reset
stx-openstack reapply will be auto-triggered, wait for it to fail or abort
force lock/unlock controller-0
remove and apply openstack

[Ghada Khalil (gkhalil)]

screening: @George, did this TC ever pass before? Manually changing the system date can result in unintended side effects.

[Ghada Khalil (gkhalil)]

screening: marking as low priority as this is a drastic test-case and is not a likely scenario for the typical user.

[George Postolache (gpostola)]

replaced the date changing on the system with modifications to the last time the password was changed to get the password to expire and it works ok