| 2014-05-12 15:51:54 |
mouadino |
bug |
|
|
added bug |
| 2014-05-13 09:21:04 |
OpenStack Infra |
neutron: status |
New |
In Progress |
|
| 2014-05-13 09:21:04 |
OpenStack Infra |
neutron: assignee |
|
mouadino (mouadino) |
|
| 2014-06-25 15:14:19 |
mouadino |
bug task added |
|
oslo |
|
| 2014-06-25 15:19:00 |
OpenStack Infra |
oslo: status |
New |
In Progress |
|
| 2014-06-25 15:19:00 |
OpenStack Infra |
oslo: assignee |
|
mouadino (mouadino) |
|
| 2014-07-07 08:43:18 |
Jaeho Song |
bug |
|
|
added subscriber Jaeho Song |
| 2014-08-26 07:58:33 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto MURATA |
| 2014-09-02 10:27:41 |
Dr. Jens Harbott |
bug |
|
|
added subscriber Dr. Jens Rosenboom |
| 2014-09-05 19:40:16 |
Doug Hellmann |
affects |
oslo-incubator |
oslo.messaging |
|
| 2014-09-09 02:59:31 |
Koji Iida |
bug |
|
|
added subscriber Koji Iida |
| 2014-11-25 09:57:31 |
Dr. Jens Harbott |
neutron: assignee |
mouadino (mouadino) |
Dr. Jens Rosenboom (j-rosenboom-j) |
|
| 2014-11-25 09:57:36 |
Dr. Jens Harbott |
oslo.messaging: assignee |
mouadino (mouadino) |
Dr. Jens Rosenboom (j-rosenboom-j) |
|
| 2014-12-03 13:07:21 |
Mehdi Abaakouk |
neutron: status |
In Progress |
Invalid |
|
| 2014-12-03 13:07:48 |
Mehdi Abaakouk |
oslo.messaging: assignee |
Dr. Jens Rosenboom (j-rosenboom-j) |
mouad.benchchaoui (mouad-benchchaoui) |
|
| 2014-12-03 13:07:53 |
Mehdi Abaakouk |
neutron: assignee |
Dr. Jens Rosenboom (j-rosenboom-j) |
|
|
| 2014-12-03 13:08:36 |
Mehdi Abaakouk |
oslo.messaging: importance |
Undecided |
Medium |
|
| 2014-12-03 13:28:39 |
Dr. Jens Harbott |
oslo.messaging: assignee |
mouad.benchchaoui (mouad-benchchaoui) |
Dr. Jens Rosenboom (j-rosenboom-j) |
|
| 2014-12-04 14:52:31 |
OpenStack Infra |
oslo.messaging: status |
In Progress |
Fix Committed |
|
| 2015-01-27 18:59:29 |
Doug Hellmann |
oslo.messaging: status |
Fix Committed |
Fix Released |
|
| 2015-01-27 18:59:29 |
Doug Hellmann |
oslo.messaging: milestone |
|
1.6.0 |
|
| 2015-07-22 11:48:15 |
Jeep |
bug task added |
|
ubuntu |
|
| 2015-12-17 07:58:27 |
Xiang Hui |
description |
In the logs the first traceback that happen is this:
[-] Unexpected exception occurred 1 time(s)... retrying.
Traceback (most recent call last):
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 62, in inner_func
return infunc(*args, **kwargs)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 741, in _consumer_thread
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 732, in consume
@excutils.forever_retry_uncaught_exceptions
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 660, in iterconsume
try:
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 590, in ensure
def close(self):
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 531, in reconnect
# to return an error not covered by its transport
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 513, in _connect
Will retry up to self.max_retries number of times.
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 150, in reconnect
use the callback passed during __init__()
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/kombu/entity.py", line 508, in declare
self.queue_bind(nowait)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/kombu/entity.py", line 541, in queue_bind
self.binding_arguments, nowait=nowait)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/kombu/entity.py", line 551, in bind_to
nowait=nowait)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/amqp/channel.py", line 1003, in queue_bind
(50, 21), # Channel.queue_bind_ok
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/amqp/abstract_channel.py", line 68, in wait
return self.dispatch_method(method_sig, args, content)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/amqp/abstract_channel.py", line 86, in dispatch_method
return amqp_method(self, args)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
reply_code, reply_text, (class_id, method_id), ChannelError,
NotFound: Queue.bind: (404) NOT_FOUND - no exchange 'reply_8f19344531b448c89d412ee97ff11e79' in vhost '/'
Than an RPC Timeout is raised each second in all the agents
ERROR neutron.agent.l3_agent [-] Failed synchronizing routers
TRACE neutron.agent.l3_agent Traceback (most recent call last):
TRACE neutron.agent.l3_agent File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 702, in _rpc_loop
TRACE neutron.agent.l3_agent self.context, router_ids)
TRACE neutron.agent.l3_agent File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 79, in get_routers
TRACE neutron.agent.l3_agent topic=self.topic)
TRACE neutron.agent.l3_agent File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/proxy.py", line 130, in call
TRACE neutron.agent.l3_agent exc.info, real_topic, msg.get('method'))
TRACE neutron.agent.l3_agent Timeout: Timeout while waiting on RPC response - topic: "q-l3-plugin", RPC method: "sync_routers" info: "<unknown>"
This actually make the agent useless until they are all restarted.
An analyze of what's going on coming soon :) |
In the logs the first traceback that happen is this:
[-] Unexpected exception occurred 1 time(s)... retrying.
Traceback (most recent call last):
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line 62, in inner_func
return infunc(*args, **kwargs)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 741, in _consumer_thread
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 732, in consume
@excutils.forever_retry_uncaught_exceptions
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 660, in iterconsume
try:
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 590, in ensure
def close(self):
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 531, in reconnect
# to return an error not covered by its transport
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 513, in _connect
Will retry up to self.max_retries number of times.
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/impl_kombu.py", line 150, in reconnect
use the callback passed during __init__()
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/kombu/entity.py", line 508, in declare
self.queue_bind(nowait)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/kombu/entity.py", line 541, in queue_bind
self.binding_arguments, nowait=nowait)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/kombu/entity.py", line 551, in bind_to
nowait=nowait)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/amqp/channel.py", line 1003, in queue_bind
(50, 21), # Channel.queue_bind_ok
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/amqp/abstract_channel.py", line 68, in wait
return self.dispatch_method(method_sig, args, content)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/amqp/abstract_channel.py", line 86, in dispatch_method
return amqp_method(self, args)
File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/amqp/channel.py", line 241, in _close
reply_code, reply_text, (class_id, method_id), ChannelError,
NotFound: Queue.bind: (404) NOT_FOUND - no exchange 'reply_8f19344531b448c89d412ee97ff11e79' in vhost '/'
Than an RPC Timeout is raised each second in all the agents
ERROR neutron.agent.l3_agent [-] Failed synchronizing routers
TRACE neutron.agent.l3_agent Traceback (most recent call last):
TRACE neutron.agent.l3_agent File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 702, in _rpc_loop
TRACE neutron.agent.l3_agent self.context, router_ids)
TRACE neutron.agent.l3_agent File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/agent/l3_agent.py", line 79, in get_routers
TRACE neutron.agent.l3_agent topic=self.topic)
TRACE neutron.agent.l3_agent File "/opt/cloudbau/neutron-virtualenv/lib/python2.7/site-packages/neutron/openstack/common/rpc/proxy.py", line 130, in call
TRACE neutron.agent.l3_agent exc.info, real_topic, msg.get('method'))
TRACE neutron.agent.l3_agent Timeout: Timeout while waiting on RPC response - topic: "q-l3-plugin", RPC method: "sync_routers" info: "<unknown>"
This actually make the agent useless until they are all restarted.
An analyze of what's going on coming soon :)
---------------------------
[Impact]
This patch addresses an issue when a RabbitMQ cluster node goes down, OpenStack services try to reconnect to another RabbitMQ node and then re-create everything from scratch , and due to the 'auto-delete' flag is set, race condition happened between re-create and delete on Exchange, Queues, Bindings, which caused nova-compute and neutron agents are down.
[Test Case]
Note steps are for trusty-icehouse, including latest oslo.messaging library (1.3.0-0ubuntu1.2 at the time of this writing).
Deploy an OpenStack cloud w/ multiple rabbit nodes and then abruptly kill one of the rabbit nodes (e.g. sudo service rabbitmq-server stop, etc). Observe that the nova services and neutron agents do detect that the node went down and report that they are reconnected, but messages are still reporting as timed out, nova service-list/neutron agent-list still reports compute and agents as down, etc.
[Regression Potential]
None. |
|
| 2015-12-17 08:19:05 |
Xiang Hui |
attachment added |
|
trusty-juno.debdiff https://bugs.launchpad.net/neutron/+bug/1318721/+attachment/4536162/+files/trusty-juno.debdiff |
|
| 2015-12-17 08:23:17 |
Ubuntu Foundations Team Bug Bot |
tags |
|
patch |
|
| 2015-12-17 08:23:29 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Sponsors Team |
| 2015-12-17 08:26:48 |
Xiang Hui |
attachment added |
|
trusty-icehouse.debdiff https://bugs.launchpad.net/neutron/+bug/1318721/+attachment/4536166/+files/trusty-icehouse.debdiff |
|
| 2015-12-17 08:40:43 |
Xiang Hui |
attachment added |
|
neutron-trusty-icehouse.debdiff https://bugs.launchpad.net/neutron/+bug/1318721/+attachment/4536168/+files/neutron-trusty-icehouse.debdiff |
|
| 2015-12-17 10:47:12 |
Xiang Hui |
branch linked |
|
lp:ubuntu/trusty-proposed/oslo.messaging |
|
| 2015-12-17 11:04:03 |
Xiang Hui |
branch linked |
|
lp:~xianghui/ubuntu/trusty/oslo.messaging/juno-lp1318721 |
|
| 2015-12-17 11:53:59 |
Launchpad Janitor |
neutron (Ubuntu): status |
New |
Confirmed |
|
| 2015-12-17 11:53:59 |
Mathew Hodson |
affects |
ubuntu |
neutron (Ubuntu) |
|
| 2015-12-17 11:53:59 |
Mathew Hodson |
neutron (Ubuntu): importance |
Undecided |
Medium |
|
| 2015-12-17 12:22:00 |
Xiang Hui |
branch linked |
|
lp:~xianghui/ubuntu/trusty/neutron/lp1318721 |
|
| 2015-12-17 12:29:54 |
Xiang Hui |
branch linked |
|
lp:~xianghui/ubuntu/trusty/oslo.messaging/lp1318721 |
|
| 2015-12-17 12:30:13 |
Xiang Hui |
branch unlinked |
lp:ubuntu/trusty-proposed/oslo.messaging |
|
|
| 2015-12-18 11:25:05 |
Mathew Hodson |
bug task added |
|
oslo.messaging (Ubuntu) |
|
| 2015-12-18 11:25:14 |
Mathew Hodson |
oslo.messaging (Ubuntu): importance |
Undecided |
Medium |
|
| 2015-12-31 17:34:03 |
Mathew Hodson |
bug task deleted |
oslo.messaging (Ubuntu) |
|
|
| 2016-01-08 08:43:49 |
Xiang Hui |
attachment added |
|
backport-1318721-trusty-juno.debdiff https://bugs.launchpad.net/neutron/+bug/1318721/+attachment/4546091/+files/backport-1318721-trusty-juno.debdiff |
|
| 2016-01-08 08:44:31 |
Xiang Hui |
attachment added |
|
backport-1318721-trusty-icehouse.debdiff https://bugs.launchpad.net/neutron/+bug/1318721/+attachment/4546092/+files/backport-1318721-trusty-icehouse.debdiff |
|
| 2016-01-08 08:44:58 |
Xiang Hui |
attachment added |
|
backport-1318721-trusty-icehouse-neutron.debdiff https://bugs.launchpad.net/neutron/+bug/1318721/+attachment/4546093/+files/backport-1318721-trusty-icehouse-neutron.debdiff |
|
| 2016-01-20 14:55:26 |
Launchpad Janitor |
branch linked |
|
lp:~corey.bryant/ubuntu/trusty/oslo.messaging/lp1318721 |
|
| 2016-01-20 15:09:06 |
Launchpad Janitor |
branch linked |
|
lp:~corey.bryant/ubuntu/trusty/neutron/lp1318721 |
|
| 2016-01-20 15:26:37 |
Corey Bryant |
neutron (Ubuntu): status |
New |
In Progress |
|
| 2016-01-20 15:26:58 |
Corey Bryant |
bug task added |
|
oslo.messaging (Ubuntu) |
|
| 2016-01-20 15:27:12 |
Corey Bryant |
nominated for series |
|
Ubuntu Trusty |
|
| 2016-01-20 15:27:12 |
Corey Bryant |
bug task added |
|
neutron (Ubuntu Trusty) |
|
| 2016-01-20 15:27:12 |
Corey Bryant |
bug task added |
|
oslo.messaging (Ubuntu Trusty) |
|
| 2016-01-20 15:27:25 |
Corey Bryant |
neutron (Ubuntu Trusty): status |
New |
In Progress |
|
| 2016-01-20 15:27:28 |
Corey Bryant |
oslo.messaging (Ubuntu Trusty): status |
New |
In Progress |
|
| 2016-01-20 15:27:31 |
Corey Bryant |
oslo.messaging (Ubuntu): status |
New |
Invalid |
|
| 2016-01-20 15:27:40 |
Corey Bryant |
neutron (Ubuntu): status |
In Progress |
Invalid |
|
| 2016-01-20 15:27:56 |
Corey Bryant |
bug task added |
|
cloud-archive |
|
| 2016-01-20 15:28:12 |
Corey Bryant |
nominated for series |
|
cloud-archive/icehouse |
|
| 2016-01-20 15:28:12 |
Corey Bryant |
bug task added |
|
cloud-archive/icehouse |
|
| 2016-01-20 15:28:12 |
Corey Bryant |
nominated for series |
|
cloud-archive/juno |
|
| 2016-01-20 15:28:12 |
Corey Bryant |
bug task added |
|
cloud-archive/juno |
|
| 2016-01-20 15:28:23 |
Corey Bryant |
cloud-archive: status |
New |
Invalid |
|
| 2016-01-20 15:28:30 |
Corey Bryant |
cloud-archive/juno: status |
New |
In Progress |
|
| 2016-01-20 15:28:33 |
Corey Bryant |
cloud-archive/icehouse: status |
New |
In Progress |
|
| 2016-01-20 15:29:20 |
Corey Bryant |
cloud-archive/icehouse: assignee |
|
Xiang Hui (xianghui) |
|
| 2016-01-20 15:29:28 |
Corey Bryant |
cloud-archive/juno: assignee |
|
Xiang Hui (xianghui) |
|
| 2016-01-20 15:29:38 |
Corey Bryant |
neutron (Ubuntu Trusty): assignee |
|
Xiang Hui (xianghui) |
|
| 2016-01-20 15:29:43 |
Corey Bryant |
oslo.messaging (Ubuntu Trusty): assignee |
|
Xiang Hui (xianghui) |
|
| 2016-01-20 15:29:58 |
Corey Bryant |
cloud-archive/juno: importance |
Undecided |
High |
|
| 2016-01-20 15:30:02 |
Corey Bryant |
cloud-archive/icehouse: importance |
Undecided |
High |
|
| 2016-01-20 15:30:06 |
Corey Bryant |
oslo.messaging (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2016-01-20 15:30:10 |
Corey Bryant |
neutron (Ubuntu Trusty): importance |
Undecided |
High |
|
| 2016-01-21 00:55:59 |
Mathew Hodson |
bug |
|
|
added subscriber Mathew Hodson |
| 2016-01-21 13:29:24 |
James Page |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2016-01-21 14:36:29 |
Chris J Arges |
neutron (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2016-01-21 14:36:34 |
Chris J Arges |
bug |
|
|
added subscriber SRU Verification |
| 2016-01-21 14:36:40 |
Chris J Arges |
tags |
patch |
patch verification-needed |
|
| 2016-01-21 14:37:12 |
Chris J Arges |
removed subscriber Ubuntu Sponsors Team |
|
|
|
| 2016-01-22 14:12:45 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/trusty-updates/oslo.messaging |
|
| 2016-01-22 14:34:07 |
Chris J Arges |
oslo.messaging (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2016-01-22 16:22:03 |
Corey Bryant |
neutron (Ubuntu Trusty): milestone |
|
trusty-updates |
|
| 2016-01-22 16:22:09 |
Corey Bryant |
oslo.messaging (Ubuntu Trusty): milestone |
|
trusty-updates |
|
| 2016-01-27 13:31:44 |
Xiang Hui |
tags |
patch verification-needed |
patch verification-done |
|
| 2016-02-03 22:21:53 |
Launchpad Janitor |
oslo.messaging (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2016-02-03 22:22:00 |
Chris J Arges |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2016-02-10 19:25:59 |
Launchpad Janitor |
branch linked |
|
lp:~ubuntu-server-dev/neutron/icehouse |
|
| 2016-02-10 22:07:18 |
Corey Bryant |
cloud-archive/juno: status |
In Progress |
Fix Committed |
|
| 2016-02-10 22:07:22 |
Corey Bryant |
cloud-archive/icehouse: status |
In Progress |
Fix Committed |
|
| 2016-02-10 22:35:42 |
Corey Bryant |
cloud-archive/juno: status |
Fix Committed |
Fix Released |
|
| 2016-02-10 22:35:51 |
Corey Bryant |
cloud-archive/icehouse: status |
Fix Committed |
Fix Released |
|
| 2016-02-23 00:33:30 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/trusty-proposed/oslo.messaging |
|
| 2016-03-03 19:14:35 |
Launchpad Janitor |
neutron (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2016-03-05 22:48:36 |
Mathew Hodson |
cloud-archive: status |
Invalid |
Fix Released |
|
| 2016-03-05 22:50:21 |
Mathew Hodson |
oslo.messaging (Ubuntu): importance |
Undecided |
High |
|
| 2016-03-05 22:50:24 |
Mathew Hodson |
oslo.messaging (Ubuntu): status |
Invalid |
Fix Released |
|
| 2016-03-05 22:50:30 |
Mathew Hodson |
bug task deleted |
neutron (Ubuntu) |
|
|
| 2016-03-05 22:50:44 |
Mathew Hodson |
removed subscriber Mathew Hodson |
|
|
|
| 2016-03-15 10:07:45 |
Edward Hope-Morley |
tags |
patch verification-done |
patch sts-sru verification-done |
|
| 2016-11-09 12:12:27 |
Louis Bouchard |
tags |
patch sts-sru verification-done |
patch verification-done |
|
