controller-0 did not become active controller after unlock during system initial

[Peng Peng]

Brief Description
-----------------
During system initial, after config controller was done and controller was unlock, controller-0 did not become active controller. When system was trying to "source /etc/platform/openrc", error msg return like "Openstack Admin credentials can only be loaded from the active controller."

Severity
--------
Critical

Steps to Reproduce
------------------
initialize system

Expected Behavior
------------------
After system config controller, controller-0 become active controller

Actual Behavior
----------------
After system config controller, controller-0 not become active controller

Reproducibility
---------------
Reproducible

System Configuration
--------------------
all system

Lab-name: sm-3, wcp_71-75, wp_8-12

Branch/Pull Time/Commit
-----------------------
2020-07-17_00-00-00

Last Pass
---------
2020-07-16_00-00-00

Timestamp/Logs
--------------
controller-0:~$ source /etc/platform/openrc
Openstack Admin credentials can only be loaded from the active controller.

controller-0:/var/log/puppet# grep "Error" */puppet.log
2020-07-17-08-22-57_controller/puppet.log:2020-07-17T08:24:13.439 Error: 2020-07-17 08:24:13 +0000 /usr/bin/systemctl reload syslog-ng returned 1 instead of one of [0]
2020-07-17-08-22-57_controller/puppet.log:2020-07-17T08:24:13.540 Error: 2020-07-17 08:24:13 +0000 /Stage[main]/Platform::Remotelogging/Exec[syslog-ng-reload]/returns: change from notrun to 0 failed: /usr/bin/systemctl reload syslog-ng returned 1 instead of one of [0]

latest/puppet.log:2020-07-17T08:24:13.439 Error: 2020-07-17 08:24:13 +0000 /usr/bin/systemctl reload syslog-ng returned 1 instead of one of [0]
latest/puppet.log:2020-07-17T08:24:13.540 Error: 2020-07-17 08:24:13 +0000 /Stage[main]/Platform::Remotelogging/Exec[syslog-ng-reload]/returns: change from notrun to 0 failed: /usr/bin/systemctl reload syslog-ng returned 1 instead of one of [0]

Test Activity
-------------
Sanity

[Peng Peng (ppeng)]

puppet.log added at
https://files.starlingx.kube.cengn.ca/launchpad/1887961

[Ghada Khalil (gkhalil)]

stx.5.0 / critical - this was introduced by a recent commit in master which merged on 2020-07-16
https://opendev.org/starlingx/config-files/commit/b9f0d1214f0809414b84eda189cbc5c3a94c1add

[Ghada Khalil (gkhalil)]

Correction: It is suspected that the above commit is the root-cause of the sanity issue. More investigation is needed by the dev prime to fully confirm

[Ghada Khalil (gkhalil)]

Some relevant logs:
2020-07-17T08:23:44.584 ^[[0;36mDebug: 2020-07-17 08:23:44 +0000 Exec[syslog-ng-reload](provider=posix): Executing '/usr/bin/systemctl reload syslog-ng'^[[0m
2020-07-17T08:23:44.586 ^[[0;36mDebug: 2020-07-17 08:23:44 +0000 Executing: '/usr/bin/systemctl reload syslog-ng'^[[0m
2020-07-17T08:23:44.590 ^[[mNotice: 2020-07-17 08:23:44 +0000 /Stage[main]/Platform::Remotelogging/Exec[syslog-ng-reload]/returns: Job for syslog-ng.service invalid.^[[0m
2020-07-17T08:23:44.593 ^[[1;31mError: 2020-07-17 08:23:44 +0000 /usr/bin/systemctl reload syslog-ng returned 1 instead of one of [0]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/741656

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/741656
Committed: https://git.openstack.org/cgit/starlingx/config-files/commit/?id=c730387f774b702ef74384de094e1370d97501da
Submitter: Zuul
Branch: master

commit c730387f774b702ef74384de094e1370d97501da
Author: Yuxing Jiang <email address hidden>
Date: Fri Jul 17 10:53:18 2020 -0400

    Revert "Restrict access priviledge of fm-manager.log"

    Revert the previous commit to syslog-ng may cause the unlock failure.

    This reverts commit b9f0d1214f0809414b84eda189cbc5c3a94c1add.

    Partial Bug: 1887961
    Signed-off-by: Yuxing Jiang <email address hidden>

    Change-Id: I82be0bc29dedfffe29648d09bc1cfc82af08c73f
    Signed-off-by: Yuxing Jiang <email address hidden>

[Peng Peng (ppeng)]

Issue was verified on
2020-07-20_20-00-00