StarlingX

Upgrade backup failed due to barbican api not responding

Bug #1981471 reported by Junfeng Li on 2022-07-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	StarlingX	In Progress	Low	Junfeng Li

Bug Description

Brief Description
-----------------
During upgrade backup, Barbican API was not responsive. The Ansible task failed to retrieve the secret from Barbican and the upgrade failed.

Severity
--------
Minor

Steps to Reproduce
------------------
Unknown

Expected Behavior
------------------
The secret is retrieved from Barbican API

Actual Behavior
----------------
The secret retrieval failed

Reproducibility
---------------
Occasional. Seen twice.

Workaround
----------
Manually resume the upgrade from where it failed

Timestamp/Logs
--------------
// TASK [common/push-docker-images : Get the k8s_registry barbican secret if it's authenticated] ***
Sunday 26 June 2022 18:52:50 +0000 (0:00:00.033) 0:00:51.659 ***********
fatal: [localhost]: FAILED! => changed=true
  cmd: source /etc/platform/openrc; openstack secret get 0277cb66-dc76-4a3f-bb65-088f307fc83c -p -f value
  delta: '0:00:04.791836'
  end: '2022-06-26 18:52:55.355851'
  msg: non-zero return code
  rc: 1
  start: '2022-06-26 18:52:50.564015'
  stderr: |-
    5xx Server error: <html>
      <head>
        <title>Internal Server Error</title>
      </head>
      <body>
        <h1><p>Internal Server Error</p></h1>

</body>
</html>

File "/usr/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 135, in handle
    self.handle_request(listener, req, client, addr)
  File "/usr/lib/python2.7/site-packages/gunicorn/workers/sync.py", line 176, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/usr/lib/python2.7/site-packages/paste/urlmap.py", line 203, in __call__
    return app(environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1313, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_middleware/base.py", line 131, in __call__
    response = req.get_response(self.application)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1313, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/site-packages/webob/request.py", line 1277, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 129, in __call__
    resp = self.call_func(req, *args, **kw)
  File "/usr/lib/python2.7/site-packages/webob/dec.py", line 193, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 338, in __call__
    response = self.process_request(req)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 659, in process_request
    resp = super(AuthProtocol, self).process_request(request)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 411, in process_request
    allow_expired=allow_expired)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 445, in _do_fetch_token
    data = self.fetch_token(token, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 754, in fetch_token
    allow_expired=allow_expired)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_identity.py", line 180, in verify_token
    allow_expired=allow_expired)
  File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/_identity.py", line 75, in verify_token
    access_rules_support=ACCESS_RULES_SUPPORT)
  File "/usr/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 120, in validate
    access_rules_support=access_rules_support)
  File "/usr/lib/python2.7/site-packages/keystoneclient/v3/tokens.py", line 93, in get_token_data
    resp, body = self._client.get(url, headers=headers)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 386, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 545, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/adapter.py", line 248, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 906, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 1001, in _send_request
    raise exceptions.SSLError(msg)
SSLError: SSL exception connecting to https://[fd01:8d::2]:5001/v3/auth/tokens: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

See original description

Tags:

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-07-12: Fix proposed to ansible-playbooks (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/849579

Changed in starlingx:
status:	New → In Progress

Junfeng Li (jli14) on 2022-07-13

description:

updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-07-19: Change abandoned on ansible-playbooks (master)

Change abandoned by "Junfeng Li <email address hidden>" on branch: master
Review: https://review.opendev.org/c/starlingx/ansible-playbooks/+/849579
Reason: This issue will be fixed in different commit.

Ghada Khalil (gkhalil) on 2022-09-10

Changed in starlingx:
importance:	Undecided → Low
tags:	added: stx.update
Changed in starlingx:
assignee:	nobody → Junfeng Li (jli14)

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.