openstack-barbican-api.service lost port binding to IPv6 interface

[Chris Winnicki]

Brief Description
-----------------
openstack-barbican-api.service lost port binding to IPv6 interface after ansible playbook replay

Severity
--------
Major - images cannot be downloaded

Steps to Reproduce
------------------
Install controller-0

Example of localhost.yml
[sysadmin@controller-0 ~(keystone_admin)]$ cat localhost.yml
system_mode: duplex
dns_servers:
- 2620:10a:a001:a103::2

management_subnet: fd00:204::/64
management_multicast_subnet: ff05::1b:0/124

cluster_host_subnet: fd00:205::/64
cluster_pod_subnet: fd00:206::/64
cluster_service_subnet: fd00:207::/112

external_oam_subnet: 2620:10a:a001:a103::6:0/64
external_oam_gateway_address: 2620:10a:a001:a103::6:0
external_oam_floating_address: 2620:10a:a001:a103::1237
external_oam_node_0_address: 2620:10a:a001:a103::1235
external_oam_node_1_address: 2620:10a:a001:a103::1236

admin_password: admin_password
ansible_become_pass: admin_become_password
pxeboot_subnet: 192.168.202.0/24
docker_http_proxy: http://yow-proxomatic.wrs.com:3128
docker_https_proxy: http://yow-proxomatic.wrs.com:3129
docker_no_proxy:
- registry.local
- tis-lab-registry.cumulus.wrs.com
- tis-lab-docker-registry.cumulus.wrs.com

ssl_ca_cert: /home/sysadmin/ca-cert.pem
docker_registries:
  unified:
    url: tis-lab-docker-registry.cumulus.wrs.com
    username: username
    password: userpassword

* run the ansible playbook (maybe required to run it more than once)
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap/bootstrap.yml

Expected Behavior
------------------
ansible-playbook should execute to completion without errors

Actual Behavior
----------------
ansible-playbook failed with the following:

2019-09-17 18:19:40,952 p=138621 u=sysadmin | fatal: [localhost]: FAILED! => {"changed": true, "cmd": "source /etc/platform/openrc; openstack secret store -n k8s-registry-secret -p 'username:jerry password:jP@ssw0rd' -c 'Secret href' -f value", "delta": "0:00:02.852287", "end": "2019-09-17 18:19:40.931761", "msg": "non-zero return code", "rc": 1, "start": "2019-09-17 18:19:38.079474", "stderr": "Failed to contact the endpoint at http://[abcd:204::2]:9311 for discovery. Fallback to using that endpoint as the base url.\nUnable to establish connection to http://[abcd:204::2]:9311/secrets/: HTTPConnectionPool(host='abcd:204::2', port=9311): Max retries exceeded with url: /secrets/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb26afe44d0>: Failed to establish a new connection: [Errno 111] Connection refused',))", "stderr_lines": ["Failed to contact the endpoint at http://[abcd:204::2]:9311 for discovery. Fallback to using that endpoint as the base url.", "Unable to establish connection to http://[abcd:204::2]:9311/secrets/: HTTPConnectionPool(host='abcd:204::2', port=9311): Max retries exceeded with url: /secrets/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb26afe44d0>: Failed to establish a new connection: [Errno 111] Connection refused',))"], "stdout": "", "stdout_lines": []}
2019-09-17 18:19:40,953 p=138621 u=sysadmin | PLAY RECAP *************************************************************************************************************************************************************************
2019-09-17 18:19:40,953 p=138621 u=sysadmin | localhost : ok=109 changed=28 unreachable=0 failed=1

Reproducibility
---------------
seen once. not sure how reproducible the issue is.

System Configuration
--------------------
AIO-duplex
Wind River Lab: yow-cgcs-wildcat-78_79

Branch/Pull Time/Commit
-----------------------
2019-09-16_20-00-00

Last Pass
---------
2019-09-03_20-00-00

Timestamp/Logs
--------------
2019-09-17 20:35:13
* Logs attached

Test Activity
-------------
System install

[Ghada Khalil (gkhalil)]

Marking as stx.3.0 / medium priority - until further investigation.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/684428

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/684428
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=22ded2458fb9b97ec8d38c55cc645d8f0a481f89
Submitter: Zuul
Branch: master

commit 22ded2458fb9b97ec8d38c55cc645d8f0a481f89
Author: Alex Kozyrev <email address hidden>
Date: Tue Sep 24 17:00:32 2019 -0400

    Provide Barbican with IPv6 address during bootstrap

    puppet-barbican has IPv4 address (0.0.0.0) as a default value
    for Barbican API port binding. It is possible during bootstrap
    that no proper IP address is yet assinged to mgmt network.
    Falling back to IPv4 address causes an issue on IPv6 systems.
    Need to set IPv6 address ([::]) as a default during bootstrap.

    Change-Id: I4cf432ce9f54214a0005e1784a2ed45969070f97
    Closes-Bug: 1844572
    Signed-off-by: Alex Kozyrev <email address hidden>

[Chris Winnicki (chriswinnicki)]

This issue is no longer reproducible.
Removing stx.retestneeded tag.