oslo.messaging does not honor rpc timeout and fails late
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Mirantis OpenStack |
Fix Released
|
Medium
|
Dmitry Mescheryakov | ||
Bug Description
Version: 9.x
Steps to reproduce:
1. Deploy a MOS env with 3 controllers and 1 compute node
2. Download that file and save it as simulator.py: http://
That is a modified copy of upstream simulator, if you are curious, make a diff against https:/
3. Go to compute node and apply that patch http://
4. In that console set the following variable:
RABBIT_
Populate user, pass and node_x_ip using the following parameters from /etc/nova/
5. Run
python simulator.py --url $RABBIT_URL rpc-server -w 1
6. Open another console to controller, which IP goes first in RABBIT_URL list.
7. Open yet another console to the compute node and populate RABBIT_URL variable here as well.
8. Here run
python simulator.py --url $RABBIT_URL rpc-client --timeout 10 -m 2 -w 10
With that command simulator will send 2 messages (-m) with timeout set to 10 seconds (--timeout) and interval between messages 10 seconds (-w)
9. Wait for simulator to send the first message and receives response from rpc-server. It is done once the following lines appear in console:
2017-05-05 14:05:22,129 DEBUG oslo_messaging.
2017-05-05 14:05:23,153 DEBUG oslo_messaging.
10. Once you see these lines, quickly (you have 10 seconds to do that) switch to controller console opened in step #6 and here execute
iptables -I OUTPUT 1 -p tcp --sport 5673 -j DROP
That will block AMQP traffic to that node.
11. Observe the following lines next:
2017-05-05 14:11:10,076 DEBUG oslo_messaging.
2017-05-05 14:12:10,087 DEBUG oslo.messaging.
What goes next is of no importance. Important here is that it takes oslo.messaging 60 seconds to surrender first attempt and try to reconnect while the timeout is 10 seconds. oslo.messaging should react much faster to the problem.
To remove iptables rule set in step #10 on controller execute
iptables -D OUTPUT -p tcp --sport 5673 -j DROP
That bug is very similar to https:/
| tags: | added: area-oslo |
| tags: | added: customer-found |
| Changed in mos: | |
| status: | New → Confirmed |
| importance: | Undecided → Medium |
| assignee: | nobody → Dmitry Mescheryakov (dmitrymex) |
| milestone: | none → 9.x-updates |
| description: | updated |
| description: | updated |
| description: | updated |
| Dmitry Mescheryakov (dmitrymex) wrote | #1 |
| Changed in mos: | |
| status: | Confirmed → In Progress |
| Changed in mos: | |
| milestone: | 9.x-updates → 9.2-mu-3 |
| description: | updated |
| Fuel Devops McRobotson (fuel-devops-robot) wrote Fix merged to openstack/oslo.messaging (9.0/mitaka) | #2 |
| Changed in mos: | |
| status: | In Progress → Fix Committed |
| Fuel Devops McRobotson (fuel-devops-robot) wrote Fix proposed to openstack/oslo.messaging (mcp/1.0/mitaka) | #3 |
| Fuel Devops McRobotson (fuel-devops-robot) wrote Fix merged to openstack/oslo.messaging (mcp/1.0/mitaka) | #4 |
| Ilya Bumarskov (ibumarskov) wrote | #5 |
| Changed in mos: | |
| status: | Fix Committed → Fix Released |
| Ilya Bumarskov (ibumarskov) wrote | #6 |
The fix is on review here: https:/