AIO-DX: controller-1 is in locked/offline state when vlans are configured in ipv4 system

[Peng Peng]

Brief Description
-----------------
DM config controller-1 failed. controller-1 in in locked/offline status and INSYNC status is failed.
controller-1 is not able to login using sysadmin/Li69nux*

Severity
--------
Major

Steps to Reproduce
------------------
install DX system

TC-name:

Expected Behavior
------------------
install DX success

Actual Behavior
----------------
install DX failed

Reproducibility
---------------
This is the first time saw this issue

System Configuration
--------------------
Two node system

Lab-name: SM_5-6
Branch/Pull Time/Commit
-----------------------
20231105T070103Z

Last Pass
---------
220231029T060059Z

Timestamp/Logs
--------------
[2023-11-06 16:52:16,701] 349 DEBUG MainThread ssh.send :: Send 'kubectl get hosts -n=deployment -o=wide'
[2023-11-06 16:52:16,751] 551 DEBUG MainThread ssh.exec_cmd:: Expecting \[.*@controller\-[01] .*\(keystone_admin\)\]\$ in prompt
[2023-11-06 16:52:16,822] 471 DEBUG MainThread ssh.expect :: Output:
NAME ADMINISTRATIVE OPERATIONAL AVAILABILITY PROFILE INSYNC SCOPE RECONCILED
controller-0 unlocked enabled available controller-0-profile true bootstrap true
controller-1 locked disabled offline controller-0-profile false bootstrap false

[sysadmin@controller-0 ~(keystone_admin)]$ system host-list
+----+--------------+-------------+----------------+-------------+--------------+
| id | hostname | personality | administrative | operational | availability |
+----+--------------+-------------+----------------+-------------+--------------+
| 1 | controller-0 | controller | unlocked | enabled | available |
| 2 | controller-1 | controller | locked | disabled | offline |
+----+--------------+-------------+----------------+-------------+--------------+

Automation log:
https://128.224.150.21/auto_logs/sys_install/stx/sm_5_6/202311060936/TIS_AUTOMATION.log

Test Activity
-------------
Sanity

[Ghada Khalil (gkhalil)]

Issue introduced by recent feature code changes. Review: https://review.opendev.org/c/starlingx/stx-puppet/+/886209

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/metal/+/900526

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/metal/+/900526
Committed: https://opendev.org/starlingx/metal/commit/e616a4495deac2fee0239a6077a58fe497c6f58d
Submitter: "Zuul (22348)"
Branch: master

commit e616a4495deac2fee0239a6077a58fe497c6f58d
Author: Teresa Ho <email address hidden>
Date: Wed Nov 8 10:31:13 2023 -0500

    Use FQDN for MGMT network in kickstart

    With the introduction of FQDN for MGMT network feature, the DNS lookup
    of 'controller' resolves to 'controller.internal'.
    The kickstart script uses the DNS lookup of controller to determine
    whether the system is using a IPv6 or IPv4 which results in a string
    instead of IP address or 0 return code. This causes a problem in
    installing nodes in IPv4 when the management interface is configured
    over vlan.

    The fix is to use the FQDN controller.internal.

    Test plan:
    PASS: Install IPv4 AIO-DX with mgmt vlan
    PASS: Install IPv6 AIO-DX with mgmt vlan

    Story: 2010722
    Task: 48682
    Closes-Bug: 2042953

    Signed-off-by: Teresa Ho <email address hidden>
    Change-Id: I5377587c8bc8c62a62f03123cabef7366df3dd94