Installation failure if controller-0 unlocked with management on loopback interface

[Bart Wensley]

Brief Description
-----------------
When installing a standard (2+2) system with ansible, if controller-0 is unlocked while the management network is assigned to the loopback interface, the installation cannot be completed. No subsequent nodes can be brought up (since there is no management interface to pxeboot from) and the management network cannot be moved to a physical interface (since the controller cannot be locked).

Severity
--------
Major: controller-0 must be re-installed to recover

Steps to Reproduce
------------------
Install controller-0 (standard config) and unlock it before moving the management network to a physical interface.

Expected Behavior
------------------
The solution is probably to allow controller-0 to be locked in this specific case so the management network can be moved to a physical interface.

Actual Behavior
----------------
See above

Reproducibility
---------------
Reproducible

System Configuration
--------------------
Mult-node system

Branch/Pull Time/Commit
-----------------------
SW_VERSION="19.01"
BUILD_TARGET="Unknown"
BUILD_TYPE="Informal"
BUILD_ID="n/a"

JOB="n/a"
BUILD_BY="bwensley"
BUILD_NUMBER="n/a"
BUILD_HOST="yow-bwensley-lx-vm2"
BUILD_DATE="2019-05-21 14:19:25 -0500"

BUILD_DIR="/"
WRS_SRC_DIR="/localdisk/designer/bwensley/starlingx-1/cgcs-root"
WRS_GIT_BRANCH="HEAD"
CGCS_SRC_DIR="/localdisk/designer/bwensley/starlingx-1/cgcs-root/stx"
CGCS_GIT_BRANCH="HEAD"

Last Pass
---------
Never

Timestamp/Logs
--------------
N/A

Test Activity
-------------
Developer Testing

[Ghada Khalil (gkhalil)]

Did you follow the instructions to update the mgmt interface?

system host-if-delete controller-0 lo
system host-if-modify -n mgmt -c platform --networks mgmt controller-0 eno2
system host-if-modify -m 9216 -n clusterhst -c platform --networks cluster-host controller-0 ens1f1
system host-if-modify -n oam -c platform --networks oam controller-0 eno1

[Bart Wensley (bartwensley)]

I mistakenly unlocked controller-0 without following those instructions (since those instructions do not exist on the installation wiki). Once I did that, it is impossible to recover without a re-install.

[Ghada Khalil (gkhalil)]

Given the system didn't recover, marking as release gating. Suggest to add a semantic check to ensure the mgmt and cluster interfaces are assigned to a valid interface

[Tee Ngo (teewrs)]

Like this LP :). In general, we certainly need to reject host-unlock request if the required controller-0 configuration steps (post Ansible bootstrap) have not been carried out.

I finally got the green light to update the installation wikis today. They have been updated. Prior to that, the instructions had been sent to StarlingX doc team, PV and the community via email on May 9th as wiki update was disallowed at that time.

[Matt Peters (mpeters-wrs)]

It is not just validating that the required interfaces are configured. The user must be able to lock controller-0 as long as it is still in a simplex state (/etc/platform/simplex exists). This is required to handle reconfiguration of any host level data, including fixing possible interface configuration issues.

[Ghada Khalil (gkhalil)]

As per discussion with Matt, changing the priority to Medium given the system config error cannot be corrected once the issue occurs.

[Ghada Khalil (gkhalil)]

As per agreement with the community, moving all unresolved medium priority bugs from stx.2.0 to stx.3.0

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/679837

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679837
Committed: https://git.openstack.org/cgit/starlingx/config/commit/?id=94d335c7e41271241ecb1b03723168f1e17755cb
Submitter: Zuul
Branch: master

commit 94d335c7e41271241ecb1b03723168f1e17755cb
Author: Teresa Ho <email address hidden>
Date: Wed Aug 21 10:27:32 2019 -0400

    Allow locking simplex controller

    After installing a controller, if it is unlocked before reconfiguring
    the required interfaces, the installation cannot be completed.
    This commit to allow locking the controller while it is in simplex
    state. This commit also reject unlocking controller in a duplex system
    if the management and cluster-host interfaces are on the loopback
    interface.

    After making the active controller on a duplex system return to a
    simplex state by locking and removing a standby controller, the address
    of the cluster-host interface was not unallocated properly. This commit
    corrected this problem.

    Closes-Bug: 1830082

    Change-Id: I8cb5d1dfd7d4ba73f40aefe3242ca32d91b4e7e8
    Signed-off-by: Teresa Ho <email address hidden>