nova-conductor is having problem with rabbitmq

[Dincer Celik]

Using stable/stein on Ubuntu 18.04 and having error while creating new instance. This is not a network or firewall related issue because all other services work like a charm. I was having the same issue with nova-api and eventlet monkey patching fixed it. I'm not sure this is related with it too.

2019-09-16 17:24:14.716 6 ERROR oslo.messaging._drivers.impl_rabbit [req-b8faac40-ef24-4c54-a2c7-dd0cd325821d 266e838cc8dc472ead771dcfb15c08b7 e639622000cc419486083d784015215b - default default] Connection failed: [Errno 111] ECONNREFUSED (retrying in 32.0 seconds): ConnectionRefusedError: [Errno 111] ECONNREFUSED

[Ken Giusti (kgiusti)]

Could this be the issue fixed in:

https://opendev.org/openstack/oslo.messaging/commit/7b8fd6370c8b04c3836ee4f9d06eaa90c7be5197

Looks like it merged on stable/stein but I don't think there's been a release for it.

[Herve Beraud (herveberaud)]

Did you use a rabbitmq cluster?

If yes to my previous question then, can you take a look if all your cluster nodes are up during this period?

The commit related to Ken's comment fix an issue in a specific situation where the socket is closed and the connection was badly ensured and redispatched...

[Dincer Celik (dincercelik)]

Yes rabbitmq is cluster but the problem is not there. I've also tried that patch before and it didn't work. My current workaround is disabling the heartbeats for nova.

[oslo_messaging_rabbit]
heartbeat_timeout_threshold = 0

[Matt Riedemann (mriedem)]

@Dincer: is this proposed nova release note for stable/stein relevant to you at all?

https://review.opendev.org/#/c/662095/4/releasenotes/notes/eventlet-monkey-patch-5f734ef581aa550e.yaml

Maybe only the part about eventlet monkey patching and the API?

[Dincer Celik (dincercelik)]

@Matt: this release note refers to my problem and setting OS_NOVA_DISABLE_EVENTLET_PATCHING=yes fixed the problem.

[farsheed ahmadian (farshan)]

this issue not related to rabbitmq, i faced with this problem and fix is using this document:

https://docs.openstack.org/nova/2024.1/admin/configuration/service-user-token.html#service-user-token

add this configs in controller node: nova.conf, neutron.conf, cinder.conf
add this configs in compute node: nova.conf
add this configs in block storage node: cinder.conf

[keystone_authtoken]
#...
service_token_roles = Role_assigned_to_cinder_nova_neutron_like_admin
service_token_roles_required = true

[service_user]
send_service_user_token = true
auth_url = http://controller:5000/identity
auth_strategy = keystone
auth_type = password
project_domain_name = Default
project_name = service
user_domain_name = Default
username = nova
password = NOVA_PASS

and consider this is important in *.conf file using transport_url add raabitmq port:
by default port is 5672
ex: transport_url = rabbit://openstack:RABBIT_PASS@controller:5672/

then you have to restart all services you changed .conf file, or reboot each node