I have a openstack environment deployed by Kolla-ansible:rocky-2023-1.
About three months ago, I inspect my baremetal node in-band by pxe and ipmi successfully, the other day, I pull new image through quay.io and inspect my baremetal node, it's failed immediately. Nothing has changed in my cluster environment, the only thing that changed was the sha256 of the ironic and ironic-inspector images.
There're the logs of inspector.log
2024-04-08 23:29:03.752 7 ERROR ironic_inspector.utils [None req-db38b1f6-cbce-41d8-9a31-d55a07009f62 c2add0cca75148ec8bbf642d44c07d12 f8632392de6c40c388f56da93cd84069 - - default default] The requested node was not found.: sqlalchemy.exc.NoResultFound: No row was found when one was required
2024-04-08 23:29:03.753 7 ERROR ironic_inspector.utils [None req-db38b1f6-cbce-41d8-9a31-d55a07009f62 c2add0cca75148ec8bbf642d44c07d12 f8632392de6c40c388f56da93cd84069 - - default default] Could not find node 963e9e22-e9f8-412e-bee4-5348d98a4b6e in cache: ironic_inspector.utils.NodeNotFoundInDBError: The requested node was not found.
2024-04-08 23:29:03.755 7 INFO eventlet.wsgi.server [None req-db38b1f6-cbce-41d8-9a31-d55a07009f62 c2add0cca75148ec8bbf642d44c07d12 f8632392de6c40c388f56da93cd84069 - - default default] 192.168.100.1,192.168.100.12 "GET /v1/introspection/963e9e22-e9f8-412e-bee4-5348d98a4b6e HTTP/1.1" status: 404 len: 316 time: 0.0387254
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main [None req-28a3a66c-61a9-40f6-857b-f36697020b96 c2add0cca75148ec8bbf642d44c07d12 f8632392de6c40c388f56da93cd84069 - - default default] Internal server error: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 919880da0aaf44ae9610d3d17cf4f9ea
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main Traceback (most recent call last):
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 441, in get
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main return self._queues[msg_id].get(block=True, timeout=timeout)
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/usr/lib64/python3.9/queue.py", line 179, in get
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main raise Empty
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main _queue.Empty
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main During handling of the above exception, another exception occurred:
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main Traceback (most recent call last):
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/ironic_inspector/main.py", line 186, in wrapper
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main return func(*args, **kwargs)
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/ironic_inspector/main.py", line 303, in wrapper
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main return func(*args, **kwargs)
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/ironic_inspector/main.py", line 379, in api_introspection
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main client.call({}, 'do_introspection', node_id=node_id,
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/client.py", line 546, in call
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main return self.prepare().call(ctxt, method, **kwargs)
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/rpc/client.py", line 190, in call
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main result = self.transport._send(
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main return self._driver.send(target, ctxt, message,
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main return self._send(target, ctxt, message, wait_for_reply, timeout,
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 678, in _send
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main result = self._waiter.wait(msg_id, timeout,
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 567, in wait
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main message = self.waiters.get(msg_id, timeout=timeout)
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 443, in get
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main raise oslo_messaging.MessagingTimeout(
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 919880da0aaf44ae9610d3d17cf4f9ea
2024-04-08 23:31:30.879 7 ERROR ironic_inspector.main
2024-04-08 23:31:30.893 7 INFO eventlet.wsgi.server [None req-28a3a66c-61a9-40f6-857b-f36697020b96 c2add0cca75148ec8bbf642d44c07d12 f8632392de6c40c388f56da93cd84069 - - default default] Traceback (most recent call last):
File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/wsgi.py", line 605, in handle_one_response
write(b''.join(towrite))
File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/wsgi.py", line 533, in write
wfile.flush()
File "/usr/lib64/python3.9/socket.py", line 722, in write
return self._sock.send(b)
File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/greenio/base.py", line 401, in send
return self._send_loop(self.fd.send, data, flags)
File "/var/lib/kolla/venv/lib64/python3.9/site-packages/eventlet/greenio/base.py", line 388, in _send_loop
return send_method(data, *args)
ConnectionResetError: [Errno 104] Connection reset by peer
2024-04-08 23:31:30.894 7 INFO eventlet.wsgi.server [None req-28a3a66c-61a9-40f6-857b-f36697020b96 c2add0cca75148ec8bbf642d44c07d12 f8632392de6c40c388f56da93cd84069 - - default default] 192.168.100.1,192.168.100.12 "POST /v1/introspection/963e9e22-e9f8-412e-bee4-5348d98a4b6e HTTP/1.1" status: 500 len: 0 time: 180.6393862
2024-04-08 23:31:30.896 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [0180f47b-7f76-4a04-84e8-ae4f3e6a5219] AMQP server on 192.168.100.12:5672 is unreachable: [Errno 104] Connection reset by peer. Trying again in 1 seconds.: ConnectionResetError: [Errno 104] Connection reset by peer
2024-04-08 23:31:31.746 7 ERROR ironic_inspector.utils [-] The requested node was not found.: sqlalchemy.exc.NoResultFound: No row was found when one was required
2024-04-08 23:31:31.828 7 INFO ironic_inspector.introspect [-] [node: 963e9e22-e9f8-412e-bee4-5348d98a4b6e state starting] The following attributes will be used for look up: {'bmc_address': ['192.168.255.7']}
2024-04-08 23:31:37.733 7 INFO ironic_inspector.introspect [-] [node: 963e9e22-e9f8-412e-bee4-5348d98a4b6e state starting] Introspection started successfully
2024-04-08 23:31:37.734 7 INFO ironic_inspector.node_cache [-] [node: 963e9e22-e9f8-412e-bee4-5348d98a4b6e state starting] Updating node state: starting --> waiting
I can't get my old image back from quay.io
When I downgraded to zed, Irony-Inspector was able to inspect the node successfully again