When performing a heartbeat to rabbit (inside a nova-compute process), there is a greenlet error which causes a hard crash.
I'm not exactly sure what details are relevant, but can provide more info if there's something that will be useful!
This is on RHEL7 (essentially... somewhat custom image based on it)
Log snippet:
``` 2021-07-07 19:34:52,686 DEBUG [oslo.messaging._drivers.impl_rabbit] /opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/oslo_messaging/_drivers/impl_rabbit.py:__init__:608 [279fc413-9d7c-4fad-89e8-8de308658947] Connecting to AMQP server on 127.0.0.1:5671 2021-07-07 19:34:52,699 DEBUG [amqp.connection.Connection.heartbeat_tick] /opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/amqp/connection.py:heartbeat_tick:726 heartbeat_tick : for connection 79f7cf4331b34cb0a2e3608281076773 2021-07-07 19:34:52,699 DEBUG [amqp.connection.Connection.heartbeat_tick] /opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/amqp/connection.py:heartbeat_tick:740 heartbeat_tick : Prev sent/recv: None/None, now - 6/6, monotonic - 9634.717472491, last_heartbeat_sent - 9634.717470288, heartbeat int. - 60 for connection 79f7cf4331b34cb0a2e3608281076773 2021-07-07 19:34:52,700 DEBUG [amqp.connection.Connection.heartbeat_tick] /opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/amqp/connection.py:heartbeat_tick:726 heartbeat_tick : for connection 79f7cf4331b34cb0a2e3608281076773 2021-07-07 19:34:52,701 DEBUG [amqp.connection.Connection.heartbeat_tick] /opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/amqp/connection.py:heartbeat_tick:740 heartbeat_tick : Prev sent/recv: 6/6, now - 6/6, monotonic - 9634.719438155, last_heartbeat_sent - 9634.717470288, heartbeat int. - 60 for connection 79f7cf4331b34cb0a2e3608281076773 2021-07-07 19:34:52,718 DEBUG [amqp] /opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/amqp/connection.py:_on_start:382 Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': '<email address hidden>', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 23.0.2', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN', b'EXTERNAL'], locales: ['en_US'] 2021-07-07 19:34:52,719 DEBUG [amqp] /opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/amqp/channel.py:__init__:104 using channel_id: 1 2021-07-07 19:34:52,720 DEBUG [amqp] /opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/amqp/channel.py:_on_open_ok:444 Channel open 2021-07-07 19:34:52,721 DEBUG [amqp.connection.Connection.heartbeat_tick] /opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/amqp/connection.py:heartbeat_tick:726 heartbeat_tick : for connection c0299792d20e42a2b0a17d037d7d3058 Traceback (most recent call last): File "/opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/eventlet/hubs/hub.py", line 476, in fire_timers timer() File "/opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/eventlet/hubs/timer.py", line 59, in __call__ cb(*args, **kw) File "/opt/openstack/venv/nova-23.0.20031033070/lib/python3.8/site-packages/eventlet/semaphore.py", line 152, in _do_acquire waiter.switch() greenlet.error: cannot switch to a different thread ```
Versions:
``` oslo.messaging==12.7.1 nova==23.0.2 (packaged locally from stable/wallaby as of July 3, 2021) ```
When performing a heartbeat to rabbit (inside a nova-compute process), there is a greenlet error which causes a hard crash.
I'm not exactly sure what details are relevant, but can provide more info if there's something that will be useful!
This is on RHEL7 (essentially... somewhat custom image based on it)
Log snippet:
``` ._drivers. impl_rabbit] /opt/openstack/ venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ oslo_messaging/ _drivers/ impl_rabbit. py:__init_ _:608 [279fc413- 9d7c-4fad- 89e8-8de3086589 47] Connecting to AMQP server on 127.0.0.1:5671 n.Connection. heartbeat_ tick] /opt/openstack/ venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ amqp/connection .py:heartbeat_ tick:726 heartbeat_tick : for connection 79f7cf4331b34cb 0a2e36082810767 73 n.Connection. heartbeat_ tick] /opt/openstack/ venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ amqp/connection .py:heartbeat_ tick:740 heartbeat_tick : Prev sent/recv: None/None, now - 6/6, monotonic - 9634.717472491, last_heartbeat_sent - 9634.717470288, heartbeat int. - 60 for connection 79f7cf4331b34cb 0a2e36082810767 73 n.Connection. heartbeat_ tick] /opt/openstack/ venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ amqp/connection .py:heartbeat_ tick:726 heartbeat_tick : for connection 79f7cf4331b34cb 0a2e36082810767 73 n.Connection. heartbeat_ tick] /opt/openstack/ venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ amqp/connection .py:heartbeat_ tick:740 heartbeat_tick : Prev sent/recv: 6/6, now - 6/6, monotonic - 9634.719438155, last_heartbeat_sent - 9634.717470288, heartbeat int. - 60 for connection 79f7cf4331b34cb 0a2e36082810767 73 venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ amqp/connection .py:_on_ start:382 Start from server, version: 0.9, properties: {'capabilities': {'publisher_ confirms' : True, 'exchange_ exchange_ bindings' : True, 'basic.nack': True, 'consumer_ cancel_ notify' : True, 'connection. blocked' : True, 'consumer_ priorities' : True, 'authentication _failure_ close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': '<email address hidden>', 'copyright': 'Copyright (c) 2007-2020 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 1.1. Website: https:/ /rabbitmq. com', 'platform': 'Erlang/OTP 23.0.2', 'product': 'RabbitMQ', 'version': '3.8.5'}, mechanisms: [b'PLAIN', b'AMQPLAIN', b'EXTERNAL'], locales: ['en_US'] venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ amqp/channel. py:__init_ _:104 using channel_id: 1 venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ amqp/channel. py:_on_ open_ok: 444 Channel open n.Connection. heartbeat_ tick] /opt/openstack/ venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ amqp/connection .py:heartbeat_ tick:726 heartbeat_tick : for connection c0299792d20e42a 2b0a17d037d7d30 58 /venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ eventlet/ hubs/hub. py", line 476, in fire_timers /venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ eventlet/ hubs/timer. py", line 59, in __call__ /venv/nova- 23.0.2003103307 0/lib/python3. 8/site- packages/ eventlet/ semaphore. py", line 152, in _do_acquire
2021-07-07 19:34:52,686 DEBUG [oslo.messaging
2021-07-07 19:34:52,699 DEBUG [amqp.connectio
2021-07-07 19:34:52,699 DEBUG [amqp.connectio
2021-07-07 19:34:52,700 DEBUG [amqp.connectio
2021-07-07 19:34:52,701 DEBUG [amqp.connectio
2021-07-07 19:34:52,718 DEBUG [amqp] /opt/openstack/
2021-07-07 19:34:52,719 DEBUG [amqp] /opt/openstack/
2021-07-07 19:34:52,720 DEBUG [amqp] /opt/openstack/
2021-07-07 19:34:52,721 DEBUG [amqp.connectio
Traceback (most recent call last):
File "/opt/openstack
timer()
File "/opt/openstack
cb(*args, **kw)
File "/opt/openstack
waiter.switch()
greenlet.error: cannot switch to a different thread
```
Versions:
``` =12.7.1
oslo.messaging=
nova==23.0.2 (packaged locally from stable/wallaby as of July 3, 2021)
```