oslo.messaging

  1. Bug #1789177
  2. Comment #58

Comment 58 for bug 1789177

Revision history for this message
Edward Hope-Morley (hopem) wrote : Re: RabbitMQ fails to synchronize exchanges under high load

e.g. 2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to process message ... skipping it.: DuplicateMessageError: Found duplicate message(fc9335298407444ab0e7000d3fe2f4b7). Skipping it.
2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 368, in _callback
2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit self.callback(RabbitMessage(message))
2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in __call__
2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit unique_id = self.msg_id_cache.check_duplicate_message(message)
2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqp.py", line 121, in check_duplicate_message
2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit raise rpc_common.DuplicateMessageError(msg_id=msg_id)
2021-02-02 12:07:53.930 27349 ERROR oslo.messaging._drivers.impl_rabbit DuplicateMessageError: Found duplicate message(fc9335298407444ab0e7000d3fe2f4b7). Skipping it.

and

2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-e53cf710-52f8-4790-bb7a-9968807f842f - - - - -] Error while processing VIF ports: MessagingTimeout: Timed out waiting for a reply to message ID 06bc2386bc6b42f2ad48ebc615
7b3ec6
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2163, in rpc_loop
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent port_info, provisioning_needed)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 158, in wrapper
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = f(*args, **kwargs)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1740, in process_network_ports
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent failed_devices['added'] |= self._bind_devices(need_binding_devices)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 892, in _bind_devices
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.conf.host, agent_restarted=agent_restarted)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/agent/rpc.py", line 165, in update_device_list
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent agent_restarted=agent_restarted)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 185, in call
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent time.sleep(wait)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent self.force_reraise()
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent six.reraise(self.type_, self.value, self.tb)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 162, in call
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent return self._original_context.call(ctxt, method, **kwargs)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 174, in call
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=self.retry)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 131, in _send
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent timeout=timeout, retry=retry)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in send
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent retry=retry)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 548, in _send
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent result = self._waiter.wait(msg_id, timeout)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 440, in wait
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent message = self.waiters.get(msg_id, timeout=timeout)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 328, in get
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent 'to message ID %s' % msg_id)
2021-02-02 12:05:54.869 27349 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID 06bc2386bc6b42f2ad48ebc6157b3ec6