Random OpenStackApiAuthenticationException: Authentication error in nova functional tests

[Matt Riedemann]

Seen here when trying to create a flavor:

http://logs.openstack.org/11/485011/4/gate/gate-nova-tox-functional-py35-ubuntu-xenial/42d69de/console.html#_2017-07-21_14_55_31_535188

2017-07-21 14:55:31.534787 | Captured traceback:
2017-07-21 14:55:31.534797 | ~~~~~~~~~~~~~~~~~~~
2017-07-21 14:55:31.534812 | b'Traceback (most recent call last):'
2017-07-21 14:55:31.534858 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api_sample_tests/test_flavor_access.py", line 46, in test_flavor_access_list'
2017-07-21 14:55:31.534871 | b' self._create_flavor()'
2017-07-21 14:55:31.534915 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api_sample_tests/test_flavor_access.py", line 40, in _create_flavor'
2017-07-21 14:55:31.534926 | b' subs)'
2017-07-21 14:55:31.534965 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api_samples_test_base.py", line 502, in _do_post'
2017-07-21 14:55:31.534987 | b' return self._get_response(url, method, body, headers=headers)'
2017-07-21 14:55:31.535028 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api_samples_test_base.py", line 482, in _get_response'
2017-07-21 14:55:31.535047 | b' headers=headers, strip_version=strip_version)'
2017-07-21 14:55:31.535085 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 160, in api_request'
2017-07-21 14:55:31.535101 | b' auth_result = self._authenticate()'
2017-07-21 14:55:31.535140 | b' File "/home/jenkins/workspace/gate-nova-tox-functional-py35-ubuntu-xenial/nova/tests/functional/api/client.py", line 153, in _authenticate'
2017-07-21 14:55:31.535161 | b' raise OpenStackApiAuthenticationException(response=response)'
2017-07-21 14:55:31.535188 | b'nova.tests.functional.api.client.OpenStackApiAuthenticationException: Authentication error'
2017-07-21 14:55:31.535196 | b''

Since we use noauth middleware this shouldn't be a problem.

[Matt Riedemann (mriedem)]

This could be related:

https://github.com/openstack/nova/blob/master/nova/api/openstack/placement/auth.py#L41

Maybe requests are getting mixed up in eventlet and going to the placement NoAuthMiddleware and we aren't providing a token:

https://github.com/openstack/nova/blob/master/nova/tests/functional/api/client.py#L142

Compared to the compute NoAuthMiddleware:

https://github.com/openstack/nova/blob/master/nova/api/openstack/auth.py#L32

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/486190

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/486193

[OpenStack Infra (hudson-openstack)]

Change abandoned by Matt Riedemann (<email address hidden>) on branch: master
Review: https://review.openstack.org/486193
Reason: cdent pointed out that while this will bypass the 401 issue, the placement noauthmiddleware would have different stuff going into the request than the compute api, and the compute api noauth middleware puts a request context into the request which the compute API code is going to depend on, so we'll pass the 401 but just fail later.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/486237

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/486190
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=068f85dc38458af122ded035d36d2451b6bd5af6
Submitter: Jenkins
Branch: master

commit 068f85dc38458af122ded035d36d2451b6bd5af6
Author: Chris Dent <email address hidden>
Date: Fri Jul 21 19:13:43 2017 +0100

    retry on authentication failure in api_client

    Due to eventlet threading weirdness, it appears that sometimes a request
    that is destined for the compute api fixture ends up on the placement
    api fixture. The latter uses a no auth middleware that 401s when it does
    not get a token.

    The change retries the auth request a few times before failing.

    If this has no effect, doing an eventlet.sleep() before the retry might
    be something worth trying.

    Change-Id: I9ca2022ca956be5ea67f4c40bed80115f2fae29d
    Related-Bug: #1705753

[Sean Dague (sdague)]

Found open reviews for this bug in gerrit, setting to In Progress.

review: https://review.openstack.org/486237 in branch: master

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/486825

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/486237
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fdf27abf7db233ca51f12e2926d78c272b54935b
Submitter: Jenkins
Branch: master

commit fdf27abf7db233ca51f12e2926d78c272b54935b
Author: Chris Dent <email address hidden>
Date: Fri Jul 21 23:01:32 2017 +0100

    [placement] Use wsgi_intercept in PlacementFixture

    Using wsgi_intercept to run the placement API in the PlacementFixture,
    instead of spawning an eventlet server. This should help with the weird
    situation where sometimes requests that are supposed to go to compute or
    placement go to the other.

    Some combination of six or wsgi-intercept is not happy with the use of
    socket_options in the keystoneauth1.session.TCPKeepAliveAdapter, so a
    MonkeyPatch is used to skip over it and use the parent Adapter.

    The change here continues to monkeypatch the report client so that we
    don't have to monkey patch keystone's auth and endpoint handling. We
    don't want that as it will also patch the nova side of things.

    Change-Id: I03b3abea152d45575ac6dde22ccbe32b6acb4646
    Related-Bug: #1705753

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/486825
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=eed6ced78776a3b9a7ada7b0c8ff74eaa376efaf
Submitter: Jenkins
Branch: master

commit eed6ced78776a3b9a7ada7b0c8ff74eaa376efaf
Author: Chris Dent <email address hidden>
Date: Tue Jul 25 02:22:56 2017 +0000

    Use wsgi-intercept in OSAPIFixture

    To lessen the amount of eventlet socket handling present in the
    functional tests, run osapi_compute via wsgi-intercept rather than
    spawning an eventlet server to run it in a "thread".

    To get this to work three main changes are made to the fixture:

    * gain access to the wsgi app via Loader().load_app. This is used
      because it involves the least amount of manipulation of CONF. The
      init_application in nova/api/openstack/wsgi_app.py tries to
      re-read configuration.

    * manually register the osapi_compute service (previously built
      in as a result of using WSGIService)

    * Override the TCPKeepAliveAdapter keystone session adapter. It's
      use of socket_options does not work will with the way that six and
      wsgi-intercept mess with the HTTPConnection class.

    The change in integrated_helpers is required because the new fixture
    must provide access to the WSGI app in a different way.

    Change-Id: Ia91fd7d7605dfb1a6f4376b1680cd26ea410d0f3
    Related-Bug: #1705753

[Matt Riedemann (mriedem)]

I haven't seen this since cdent's fixes:

https://review.openstack.org/#/q/topic:bug/1705753+(status:open+OR+status:merged)

So marking it fixed.