Ansible bootstrap/prepare-env failed with IPv6 error

[Alexandru Dimofte]

Brief Description
-----------------
StarlingX installation setup failed. Ansible bootstrap/prepare-env failure. Master branch is affected, the issue is visible on all configurations.

Severity
--------
<Critical: System/Feature is not usable due to the defect>

Steps to Reproduce
------------------
Try to install 20220113T023728Z

Expected Behavior
------------------
Stx installation should work fine.

Actual Behavior
----------------
2022-01-13 04:44:09,060 p=394696 u=sysadmin | TASK [bootstrap/prepare-env : Fail if host is unlocked or host configurations have already started] ***
2022-01-13 04:44:09,060 p=394696 u=sysadmin | Thursday 13 January 2022 04:44:09 +0000 (0:00:00.099) 0:00:02.022 ******
2022-01-13 04:44:09,080 p=394696 u=sysadmin | fatal: [localhost]: FAILED! => changed=false
  msg: |-
    "The bootstrap of host localhost has already been finalized. Please re-install before attempting to bootstrap it again."
2022-01-13 04:44:09,081 p=394696 u=sysadmin | PLAY RECAP *********************************************************************
2022-01-13 04:44:09,082 p=394696 u=sysadmin | localhost : ok=11 changed=3 unreachable=0 failed=1
2022-01-13 04:44:09,082 p=394696 u=sysadmin | Thursday 13 January 2022 04:44:09 +0000 (0:00:00.021) 0:00:02.044 ******
2022-01-13 04:44:09,082 p=394696 u=sysadmin | ===============================================================================
2022-01-13 04:44:09,082 p=394696 u=sysadmin | common/prepare-env : stat ----------------------------------------------- 0.51s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | common/validate-target : Retrieve software version number --------------- 0.25s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | bootstrap/prepare-env : Look for unmistakenly StarlingX packages -------- 0.16s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | common/validate-target : Retrieve system type --------------------------- 0.11s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | bootstrap/prepare-env : Check initial config flag ----------------------- 0.11s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | bootstrap/prepare-env : Check if bootstrap_finalized flag exists on host --- 0.10s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | bootstrap/prepare-env : Fail if any of the mandatory configurations are not defined --- 0.09s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | common/prepare-env : include_vars --------------------------------------- 0.05s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | common/validate-target : Set host software version, system type --------- 0.02s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | bootstrap/prepare-env : Set config path facts --------------------------- 0.02s
2022-01-13 04:44:09,082 p=394696 u=sysadmin | common/validate-target : set_fact --------------------------------------- 0.02s
2022-01-13 04:44:09,083 p=394696 u=sysadmin | bootstrap/prepare-env : Fail if host is unlocked or host configurations have already started --- 0.02s
2022-01-13 04:44:09,083 p=394696 u=sysadmin | common/validate-target : Set software version, system type config path facts --- 0.02s
2022-01-13 04:44:09,083 p=394696 u=sysadmin | common/prepare-env : Set SSH hostname if SSH port is not default -------- 0.02s
2022-01-13 04:44:09,083 p=394696 u=sysadmin | common/prepare-env : Gather remote SSH public key ----------------------- 0.02s
2022-01-13 04:44:09,083 p=394696 u=sysadmin | common/prepare-env : Fail if host is unreachable ------------------------ 0.02s
2022-01-13 04:44:09,083 p=394696 u=sysadmin | common/prepare-env : Add remote SSH public keys into the known_hosts ---- 0.02s
2022-01-13 04:44:09,083 p=394696 u=sysadmin | common/prepare-env : debug ---------------------------------------------- 0.02s
2022-01-13 04:44:09,083 p=394696 u=sysadmin | common/validate-target : Fail if system type is not defined ------------- 0.02s
2022-01-13 04:44:09,083 p=394696 u=sysadmin | common/validate-target : Fail if the boot address does not exist in this host (IPv6) --- 0.02s

Reproducibility
---------------
100% reproducible

System Configuration
--------------------
all configurations.

Branch/Pull Time/Commit
-----------------------
master branch image 20220113T023728Z

Last Pass
---------
20220108T034842Z

Timestamp/Logs
--------------
will be attached

Test Activity
-------------
Sanity

Workaround
----------
-

[Alexandru Dimofte (adimofte)]

Collected logs can be downloaded from here: https://files.starlingx.cengn.ca/download_file/3

[Ghada Khalil (gkhalil)]

screening: stx.7.0 / critical - issue resulting in red sanity and is recently introduced in the stx main branch

[Ghada Khalil (gkhalil)]

This issue looks to be the same as https://bugs.launchpad.net/starlingx/+bug/1957836
Assigning to the owner of that LP to confirm.

[Adriano Oliveira (aoliveir)]

Confirmed with Marquitti that this is actually not the same issue as https://bugs.launchpad.net/starlingx/+bug/1957836, this happens on the prepare-env stage, while the other happens on persist-config stage.
Besides that, the "invalid address" error that happens on the sysinv log does not occur in this issue.
The errors seen in this issue seem to be related to NFV-VIM communication exceptions on openstack component apis, such as nova.

[Alexandru Dimofte (adimofte)]

This issue is no longer visible using latest master image 20220115T042348Z.

[Ghada Khalil (gkhalil)]

I looked at the changelog for the master load where the issue is no longer seen:
http://mirror.starlingx.cengn.ca/mirror/starlingx/master/centos/flock/20220115T042348Z/outputs/CHANGELOG.txt and it does include the following revert:
./cgcs-root/stx/config 682ad57a2cccea6b2d16212ac448f06ea5337446 2022-01-14 20:27:03 +0000 Luis Eduardo Angelini Marquitti <email address hidden> Revert "Improving IPV4 validation for configs"
Review: https://review.opendev.org/c/starlingx/config/+/824757

Perhaps the failure signature is not exactly the same as https://bugs.launchpad.net/starlingx/+bug/1957836. However, there is a good chance that the issue reported here was also introduced by the same commit which was reverted.

Marking as Fix Released since the issue is now resolved.