ironic.common.exception.AgentAPIError - 401 Unauthorized: Token invalid.

[Harald Jensås]

Ironic Storyboard issue - https://storyboard.openstack.org/#!/stor/2007529

Provisioning and cleaning nodes are failing due to an Invalid Token error raised here:
https://opendev.org/openstack/ironic-python-agent/src/commit/af5f05a0ee6125d9f3c2a541ba164f768b47d2ea/ironic_python_agent/api/app.py#L217-L220

Based on release note agent-token should'nt be required by default:
https://opendev.org/openstack/ironic/raw/branch/master/releasenotes/notes/agent-token-support-0a5b5aa1585dfbb5.yaml

In this code False is returned if an agent_token is recived and self.agent_token is None.
Do we need a condititional to return True when self.agent_token_required is False in this case?
https://opendev.org/openstack/ironic-python-agent/src/commit/af5f05a0ee6125d9f3c2a541ba164f768b47d2ea/ironic_python_agent/agent.py#L234-L252

2020-04-08 02:31:27.340 7 ERROR ironic.drivers.modules.agent_client [-] Agent command clean.get_clean_steps for node f1a36f47-ddfd-442c-9c81-a47f5a3dca22 failed. Expected 2xx HTTP status code, got 401.
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor [-] Asynchronous exception: Node failed to start the first cleaning step. Exception: Agent API for node f1a36f47-ddfd-442c-9c81-a47f5a3dca22 returned HTTP status code 401 with error: 401 Unauthorized: Token invalid. for node f1a36f47-ddfd-442c-9c81-a47f
5a3dca22: ironic.common.exception.AgentAPIError: Agent API for node f1a36f47-ddfd-442c-9c81-a47f5a3dca22 returned HTTP status code 401 with error: 401 Unauthorized: Token invalid.
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 411, in heartbeat
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor self.refresh_clean_steps(task)
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 496, in refresh_clean_steps
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor agent_result = self._client.get_clean_steps(node, task.ports).get(
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/agent_client.py", line 287, in get_clean_steps
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor wait=True)
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 60, in wrapped
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor result = f(*args, **kwargs)
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor return Retrying(*dargs, **dkw).call(f, *args, **kw)
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor return attempt.get(self._wrap_exception)
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor six.reraise(self.value[0], self.value[1], self.value[2])
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor raise value
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/agent_client.py", line 131, in _command
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor error=result.get('faultstring'))
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor ironic.common.exception.AgentAPIError: Agent API for node f1a36f47-ddfd-442c-9c81-a47f5a3dca22 returned HTTP status code 401 with error: 401 Unauthorized: Token invalid.
2020-04-08 02:31:27.341 7 ERROR ironic.drivers.modules.agent_base_vendor

[Harald Jensås (harald-jensas)]

Closing, no longer see this issue. Suspect package promotion issues at the time was causing this.