After network glitch, nova-api service fails to reestablish connection
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Nova Cloud Controller Charm |
Invalid
|
Undecided
|
Unassigned | ||
nova (Ubuntu) |
Expired
|
Low
|
Unassigned | ||
nova-cloud-controller (Juju Charms Collection) |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Due to a network glitch the connection between nova-api-os-compute and rabbitmq was temporarily dropping packets (nf_conntrack was set to low). After fixing this, the network became stable again (no more dropped packets). However, nova-api-os-compute couldn't get a clean rabbitmq connection again:
2015-07-23 13:08:38.896 26976 AUDIT nova.api.
12bb569bf909441
2015-07-23 13:08:38.940 26976 ERROR oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
line 624, in ensure
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
line 717, in _consume
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
events
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
rain_events
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
vents
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
e_ok
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
end_method
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
te_method
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
ame
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
l
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.940 26976 TRACE oslo.messaging.
2015-07-23 13:08:38.967 26976 INFO oslo.messaging.
2015-07-23 13:08:38.967 26976 INFO oslo.messaging.
In this case, a nova volume-attach operation failed.
After restarting the nova-api-os-compute process the volume-attach was functional again, also no connection breakage in the log
Versions:
n-c-c charm: lp:~canonical-bootstack/charms/trusty/nova-cloud-controller/ps45-filters;revno=164
nova-api-os-compute 1:2014.
Both n-c-c and rabbitmq are containerized on the same metal
Changed in nova-cloud-controller (Juju Charms Collection): | |
status: | New → Invalid |
This is quite and old bug, but the resilience of AMQP connections in nova is really a nova/oslo.messaging problem, rather than a charm problem; raising bug tasks to this effect.
This is also quite an old openstack version - so general improvements may have been made since Kilo.