Swift Authorization Failure in tempest job

[Eric K]

The key line is: Authorization failed: (http://148.62.14.64:5000/v2.0/auth/tokens): The resource could not be found. (HTTP 404)

It may have something to do with the auth url version workaround: https://github.com/openstack/congress/blob/master/congress/datasources/swift_driver.py#L68

One thing that's strange: I don't seem to get this error in my local devstack.

Selected log (full logs attached also here for a time: http://logs.openstack.org/59/449359/5/check/gate-congress-dsvm-api-mysql-ubuntu-xenial/a6579a4/logs/):
2017-03-28 23:07:57.763 31258 INFO congress.datasources.datasource_driver [-] swift:: polling
2017-03-28 23:07:57.764 31258 DEBUG congress.datasources.datasource_driver [-] update table objects. update_from_datasource /opt/stack/new/congress/congress/datasources/datasource_driver.py:1374
2017-03-28 23:07:57.764 31258 DEBUG congress.datasources.swift_driver [-] Swift obtaining containers List _get_containers_and_objects /opt/stack/new/congress/congress/datasources/swift_driver.py:113
2017-03-28 23:07:57.791 31258 DEBUG keystoneclient.auth.identity.v3.base [-] Making authentication request to http://148.62.14.64:5000/v2.0/auth/tokens get_auth_ref /usr/local/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py:189
2017-03-28 23:07:57.804 31258 DEBUG keystoneclient.session [-] Request returned failure status: 404 request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:444
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service [-] Authorization Failure. Authorization failed: (http://148.62.14.64:5000/v2.0/auth/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-7d51b1d6-fc44-416d-be45-2e84b9cd1396)
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service Traceback (most recent call last):
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service File "/usr/local/lib/python2.7/dist-packages/swiftclient/service.py", line 892, in _list_account_job
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service headers=req_headers
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1724, in get_account
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service full_listing=full_listing, headers=headers)
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1661, in _retry
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service self.url, self.token = self.get_auth()
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1613, in get_auth
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service timeout=self.timeout)
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 669, in get_auth
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service auth_version=auth_version)
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 581, in get_auth_keystone
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service raise ClientException('Authorization Failure. %s' % err)
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service ClientException: Authorization Failure. Authorization failed: (http://148.62.14.64:5000/v2.0/auth/tokens): The resource could not be found. (HTTP 404) (Request-ID: req-7d51b1d6-fc44-416d-be45-2e84b9cd1396)
2017-03-28 23:07:57.809 31258 ERROR swiftclient.service
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver [-] Datasource driver raised exception
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver Traceback (most recent call last):
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver File "/opt/stack/new/congress/congress/datasources/datasource_driver.py", line 1388, in poll
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver self.update_from_datasource() # sets self.state
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver File "/opt/stack/new/congress/congress/datasources/datasource_driver.py", line 1375, in update_from_datasource
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver self.update_methods[registered_table]()
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver File "/opt/stack/new/congress/congress/datasources/swift_driver.py", line 105, in <lambda>
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver self._get_containers_and_objects(objects=True))
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver File "/opt/stack/new/congress/congress/datasources/swift_driver.py", line 115, in _get_containers_and_objects
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver containers = stats['listing']
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver KeyError: 'listing'
2017-03-28 23:07:57.809 31258 ERROR congress.datasources.datasource_driver
2017-03-28 23:07:57.810 31258 INFO congress.datasources.datasource_driver [-] swift:: finished polling

[Eric K (ekcs)]

Reproduced on local setup with new devstack version. Setting priority to high.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541529
Committed: https://git.openstack.org/cgit/openstack/congress/commit/?id=c2df0090189c90a4b90a2d2b1e6527bdc22ae2d3
Submitter: Zuul
Branch: master

commit c2df0090189c90a4b90a2d2b1e6527bdc22ae2d3
Author: Eric K <email address hidden>
Date: Tue Feb 6 19:23:53 2018 -0800

    Fix swift auth issue by specifying version in url

    Swift client does not always handle unversioned identity URL
    properly. Leading to authorization failure because keystone cannot
    handle incorrect path, returning 404.

    To work around the issue, congress swift driver
    specifies the version in url if none is already provided.

    Partial-Bug: 1677380

    Change-Id: I8df7dac60e4f7cb3c2ec792e13f687d5844c8409

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541531
Committed: https://git.openstack.org/cgit/openstack/congress-tempest-plugin/commit/?id=bc8ef7c931ab12c4774a34870e586ec9feae19a7
Submitter: Zuul
Branch: master

commit bc8ef7c931ab12c4774a34870e586ec9feae19a7
Author: Eric K <email address hidden>
Date: Tue Feb 6 19:25:51 2018 -0800

    Re-enable swift polling test along with fix

    Closes-Bug: 1677380

    Depends-On: I8df7dac60e4f7cb3c2ec792e13f687d5844c8409
    Change-Id: I5fd68295d2f06e75991874fdcd6dc1c6daa622e0