Neutron client will fail to re-authenticate if token expires because no credentials are provided

[Sam Betts]

Ironic Cond Logs:

2015-12-09 00:59:09.664 ERROR ironic.dhcp.neutron [-] Failed to update Neutron port e3dee560-f938-4f91-bec0-557ef6ff5a75.
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron Traceback (most recent call last):
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron File "/opt/stack/ironic/ironic/dhcp/neutron.py", line 64, in update_port_dhcp_opts
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron client.update_port(port_id, port_req_body)
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 102, in with_params
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron ret = self.function(instance, *args, **kwargs)
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 562, in update_port
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron return self.put(self.port_path % (port), body=body)
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 302, in put
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron headers=headers, params=params)
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 270, in retry_request
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron headers=headers, params=params)
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 200, in do_request
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron content_type=self.content_type())
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 173, in do_request
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron self.authenticate()
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron File "/usr/local/lib/python2.7/dist-packages/neutronclient/client.py", line 242, in authenticate
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron raise exceptions.Unauthorized(message=err_msg)
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron Unauthorized: Unknown auth strategy: None
2015-12-09 00:59:09.664 TRACE ironic.dhcp.neutron

Neutron Logs:

2015-12-09 00:59:09.661 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2015-12-09 00:59:09.662 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find token: 5c497cff5c6f467e8e49f49346a79d4d", "code": 404, "title": "Not Found"}}
2015-12-09 00:59:09.662 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2015-12-09 00:59:09.663 INFO neutron.wsgi [-] 10.86.7.185 - - [09/Dec/2015 00:59:09] "PUT /v2.0/ports/e3dee560-f938-4f91-bec0-557ef6ff5a75.json HTTP/1.1" 401 297 0.103369

Related Code:

https://github.com/openstack/ironic/blob/master/ironic/dhcp/neutron.py#L90

[Pavlo Shchelokovskyy (pshchelo)]

what was the part of node cycle you got that? deployment or cleaning? these two use different credentials to create the neutron client, so it might help to understand the root cause better.

[Pavlo Shchelokovskyy (pshchelo)]

On the other hand it seems to be the following - keystone_authtoken is configured to validate tokens via Keystone V2 api,
but the request with token might have been received with Keystone V3 issued token. AFAIK Keystone V2 has no notion of tokens issued by Keystone V3. Can you verify how the original request was made?

[Pavlo Shchelokovskyy (pshchelo)]

Can you please try and verify if this bug in reproduced with this patch? https://review.openstack.org/#/c/236982/

If on DevStack you'd also need this patch to DevStack itself to correctly set the keystone-related config options in ironic.conf
https://review.openstack.org/#/c/238554/

otherwise fill them in accordingly yourself.

[Sam Betts (sambetts)]

This has occurred during cleaning, and during deployment (nova boot and nova delete), in my driver implementation they are both running the same code for the neutron clients because its based on a network provider using https://review.openstack.org/#/c/139687/37/ironic/common/network.py to get the neutron client:

network.get_neutron_client(task.context.auth_token).

Since getting this error I've changed my driver to just call network.get_neutron_client() to force it to do a fresh authenticate and I no longer see the error from my code, however I'm still seeing it from the DHCP code.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/255885

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/255885
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=66cf3d00cbac0bb00aad46d64ec8629b4051fd7a
Submitter: Jenkins
Branch: master

commit 66cf3d00cbac0bb00aad46d64ec8629b4051fd7a
Author: Sam Betts <email address hidden>
Date: Thu Dec 10 13:18:21 2015 +0000

    Always pass keystone credentials to neutronclient

    If the token that is passed to the neutronclient has expired then the
    neutronclient will attempt to re-authenticate, when this happens the
    current implementation fails because it does not have the required
    credentials, this patch changes the _build_client code to pass
    the credentials and token to neutronclient and let it decide which to
    use.

    Change-Id: Icf426a269a41fc941cf95f9c7f279a468dad2400
    Closes-Bug: #1524779

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/ironic 5.0.0 release.