openstack-barbican-api.service lost port binding to IPv6 interface

Bug #1844572 reported by Chris Winnicki
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StarlingX
Fix Released
Medium
Alexander Kozyrev

Bug Description

Brief Description
-----------------
openstack-barbican-api.service lost port binding to IPv6 interface after ansible playbook replay

Severity
--------
Major - images cannot be downloaded

Steps to Reproduce
------------------
Install controller-0

Example of localhost.yml
[sysadmin@controller-0 ~(keystone_admin)]$ cat localhost.yml
system_mode: duplex
dns_servers:
- 2620:10a:a001:a103::2

management_subnet: fd00:204::/64
management_multicast_subnet: ff05::1b:0/124

cluster_host_subnet: fd00:205::/64
cluster_pod_subnet: fd00:206::/64
cluster_service_subnet: fd00:207::/112

external_oam_subnet: 2620:10a:a001:a103::6:0/64
external_oam_gateway_address: 2620:10a:a001:a103::6:0
external_oam_floating_address: 2620:10a:a001:a103::1237
external_oam_node_0_address: 2620:10a:a001:a103::1235
external_oam_node_1_address: 2620:10a:a001:a103::1236

admin_password: admin_password
ansible_become_pass: admin_become_password
pxeboot_subnet: 192.168.202.0/24
docker_http_proxy: http://yow-proxomatic.wrs.com:3128
docker_https_proxy: http://yow-proxomatic.wrs.com:3129
docker_no_proxy:
- registry.local
- tis-lab-registry.cumulus.wrs.com
- tis-lab-docker-registry.cumulus.wrs.com

ssl_ca_cert: /home/sysadmin/ca-cert.pem
docker_registries:
  unified:
    url: tis-lab-docker-registry.cumulus.wrs.com
    username: username
    password: userpassword

* run the ansible playbook (maybe required to run it more than once)
ansible-playbook /usr/share/ansible/stx-ansible/playbooks/bootstrap/bootstrap.yml

Expected Behavior
------------------
ansible-playbook should execute to completion without errors

Actual Behavior
----------------
ansible-playbook failed with the following:

2019-09-17 18:19:40,952 p=138621 u=sysadmin | fatal: [localhost]: FAILED! => {"changed": true, "cmd": "source /etc/platform/openrc; openstack secret store -n k8s-registry-secret -p 'username:jerry password:jP@ssw0rd' -c 'Secret href' -f value", "delta": "0:00:02.852287", "end": "2019-09-17 18:19:40.931761", "msg": "non-zero return code", "rc": 1, "start": "2019-09-17 18:19:38.079474", "stderr": "Failed to contact the endpoint at http://[abcd:204::2]:9311 for discovery. Fallback to using that endpoint as the base url.\nUnable to establish connection to http://[abcd:204::2]:9311/secrets/: HTTPConnectionPool(host='abcd:204::2', port=9311): Max retries exceeded with url: /secrets/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb26afe44d0>: Failed to establish a new connection: [Errno 111] Connection refused',))", "stderr_lines": ["Failed to contact the endpoint at http://[abcd:204::2]:9311 for discovery. Fallback to using that endpoint as the base url.", "Unable to establish connection to http://[abcd:204::2]:9311/secrets/: HTTPConnectionPool(host='abcd:204::2', port=9311): Max retries exceeded with url: /secrets/ (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb26afe44d0>: Failed to establish a new connection: [Errno 111] Connection refused',))"], "stdout": "", "stdout_lines": []}
2019-09-17 18:19:40,953 p=138621 u=sysadmin | PLAY RECAP *************************************************************************************************************************************************************************
2019-09-17 18:19:40,953 p=138621 u=sysadmin | localhost : ok=109 changed=28 unreachable=0 failed=1

Reproducibility
---------------
seen once. not sure how reproducible the issue is.

System Configuration
--------------------
AIO-duplex
Wind River Lab: yow-cgcs-wildcat-78_79

Branch/Pull Time/Commit
-----------------------
2019-09-16_20-00-00

Last Pass
---------
2019-09-03_20-00-00

Timestamp/Logs
--------------
2019-09-17 20:35:13
* Logs attached

Test Activity
-------------
System install

Revision history for this message
Chris Winnicki (chriswinnicki) wrote :
Revision history for this message
Chris Winnicki (chriswinnicki) wrote :
Revision history for this message
Ghada Khalil (gkhalil) wrote :

Marking as stx.3.0 / medium priority - until further investigation.

description: updated
Changed in starlingx:
importance: Undecided → Medium
status: New → Triaged
assignee: nobody → Alex Kozyrev (akozyrev)
tags: added: stx.3.0 stx.containers
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to stx-puppet (master)

Fix proposed to branch: master
Review: https://review.opendev.org/684428

Changed in starlingx:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to stx-puppet (master)

Reviewed: https://review.opendev.org/684428
Committed: https://git.openstack.org/cgit/starlingx/stx-puppet/commit/?id=22ded2458fb9b97ec8d38c55cc645d8f0a481f89
Submitter: Zuul
Branch: master

commit 22ded2458fb9b97ec8d38c55cc645d8f0a481f89
Author: Alex Kozyrev <email address hidden>
Date: Tue Sep 24 17:00:32 2019 -0400

    Provide Barbican with IPv6 address during bootstrap

    puppet-barbican has IPv4 address (0.0.0.0) as a default value
    for Barbican API port binding. It is possible during bootstrap
    that no proper IP address is yet assinged to mgmt network.
    Falling back to IPv4 address causes an issue on IPv6 systems.
    Need to set IPv6 address ([::]) as a default during bootstrap.

    Change-Id: I4cf432ce9f54214a0005e1784a2ed45969070f97
    Closes-Bug: 1844572
    Signed-off-by: Alex Kozyrev <email address hidden>

Changed in starlingx:
status: In Progress → Fix Released
Yang Liu (yliu12)
tags: added: stx.retestneeded
Revision history for this message
Chris Winnicki (chriswinnicki) wrote :

This issue is no longer reproducible.
Removing stx.retestneeded tag.

tags: removed: stx.retestneeded
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.