Subcloud bootstrap fails because Barbican service cannot start

[Andre Kantek]

Brief Description

The bootstrap of the subcloud fails at the task below.

TASK [bootstrap/persist-config : Create Barbican secret for k8s registry if credentials exist] ***
Wednesday 28 June 2023 21:01:34 +0000 (0:00:00.041) 0:15:10.855 ********
fatal: [subcloud1]: FAILED! => changed=true
  cmd: source /etc/platform/openrc; openstack secret store -n k8s-registry-secret -p 'username:admin password:Li69nux*' -c 'Secret href' -f value
  delta: '0:00:07.005836'
  end: '2023-06-28 21:01:42.096329'
  msg: non-zero return code
  rc: 1
  start: '2023-06-28 21:01:35.090493'
  stderr: |-
    Failed to contact the endpoint at http://127.0.0.1:9311 for discovery. Fallback to using that endpoint as the base url.
    Unable to establish connection to http://127.0.0.1:9311/secrets/: HTTPConnectionPool(host='127.0.0.1', port=9311): Max retries exceeded with url: /secrets/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffa1f9adaf0>: Failed to establish a new connection: [Errno 111] Connection refused'))
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
It can be seen in the /var/log/daemon.log the following message related to barbican service: "info Error: '' is not a valid port number.".

At /etc/barbican/barbican.conf, we can see that the port is not set:

#host_href = http://localhost:9311
host_href=http://controller:
This problem may be related to https://review.opendev.org/c/starlingx/stx-puppet/+/885586.

Severity

Critical.

Steps to Reproduce

Deploy a DC environment with 1 subcloud.

Expected Behavior

The subcloud bootstrap is completed with no error.

Actual Behavior

The subcloud bootstrap fails.

Reproducibility

100% reproducible.

System Configuration

Vbox DC with 1 subcloud.

Load info (eg: 2022-03-10_20-00-07)

sysadmin@controller-0:~$ cat /etc/build.info
SW_VERSION="23.09"
BUILD_TARGET="Unknown"
BUILD_TYPE="Informal"
BUILD_ID="n/a"JOB="n/a"
BUILD_BY="jpotal"
BUILD_NUMBER="n/a"
BUILD_HOST="stx-debian-stx-builder-64fd6dff6c-dvvn8"
BUILD_DATE="2023-06-27 21:24:23 +0000"BUILD_DIR="/localdisk/loadbuild/jpotal/stx-debian"
WRS_SRC_DIR="/localdisk/designer/jpotal/stx-debian/cgcs-root"
WRS_GIT_BRANCH="HEAD"
CGCS_SRC_DIR="/localdisk/designer/jpotal/stx-debian/cgcs-root/stx"
CGCS_GIT_BRANCH="HEAD"
Last Pass

The problem was not present in images from a week ago.

Timestamp/Logs

See collected logs.

Alarms

NA.

Test Activity

Feature Testing.

Workaround

If the line https://review.opendev.org/c/starlingx/stx-puppet/+/885586/5/puppet-manifests/src/modules/openstack/manifests/barbican.pp#b2 is manually reverted in the subcloud filesystem, the bootstrap does not fail.

[Andre Kantek (akantek)]

887306: Add L4 service ports value to bootstrap runtime hieradata | https://review.opendev.org/c/starlingx/ansible-playbooks/+/887306

Merged in https://opendev.org/starlingx/ansible-playbooks/commit/47a3a4826f48414c153ed07545a6eb66d5d34648

887372: Create SYSTEM_CONFIG in sysinv.common.platform_firewall | https://review.opendev.org/c/starlingx/config/+/887372

Merged in https://opendev.org/starlingx/config/commit/1f4a9dd8fd6b428160a0941b2286593a5a54340b