long kombu_reconnect_delay for Neutron blocks metadata
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Fuel for OpenStack |
Fix Released
|
High
|
Ivan Berezovskiy | ||
5.1.x |
Won't Fix
|
High
|
MOS Maintenance | ||
6.0.x |
Won't Fix
|
High
|
MOS Maintenance | ||
6.1.x |
Fix Committed
|
High
|
Alexey Deryugin | ||
7.0.x |
Fix Committed
|
High
|
Alexey Deryugin | ||
8.0.x |
Fix Released
|
High
|
Alexey Deryugin | ||
Mitaka |
Fix Released
|
High
|
Ivan Berezovskiy |
Bug Description
We've implemented the following workaround in Fuel 5.1.1:
https:/
Set delay to 5.0 to recover channel errors on highly loaded environments.
Actually, this fix had been made for slow environments.
It introduced to us a new issue, affecting all Mirantis OpenStack versions between 5.1.1 and 6.1.
Instance launch may fail.
It is related to this HA+RabbitMQ+Oslo Messaging bug:
https:/
So far customers reported the following errors:
=======
Error Message 1
Timed out waiting for a reply to message ID 7bc529a2d71141c
Code
500
Details
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
File "/usr/lib/
=======
Error message 2
The problem I am seeing is that when cloud-init would access the metadata service it was taking 10seconds to respond to each query which is slowing down the startup times of instances from ~30sec to over 3min.
time curl http://
i-00000046
real 0m10.289s
user 0m0.000s
sys 0m0.003s
After some troubleshooting I noticed that the the neutron-
2015-03-12 21:00:34.247 41004 ERROR oslo.messaging.
=======
Should we set kombu_reconnect
description: | updated |
tags: | added: low-hanging-fruit |
tags: | added: area-library |
no longer affects: | fuel/future |
no longer affects: | fuel/newton |
Changed in fuel: | |
assignee: | Fuel Sustaining (fuel-sustaining-team) → Networking (l23-network) |
tags: | added: on-verification |
tags: | added: on-verification |
The root cause of the issue looks to be the same as in https:/ /bugs.launchpad .net/mos/ +bug/1410797. This bug is already fixed in 6.1 release