oslo.messaging

agent notifier getting amqp NotFound exceptions propagated up to it

Bug #1705351 reported by Kevin Benton on 2017-07-19

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Kevin Benton	neutron pike-rc1 "p-rc1"
	oslo.messaging	Fix Released	Undecided	Ken Giusti

Bug Description

There appears to be a scenario where amqp NotFound exceptions are being propagated up through oslo messaging to the Neutron RPC client attempting to perform a cast.

Since this appears in a grenade job, my guess is that it's due to the agents no longer listening on that particular exchange after they have been upgraded (the "port-delete" exchange is no longer used by agents in master).

In neutron we probably need to ignore these exceptions on casts since it just means there are no longer agents interested in listening to the casts. However, this also seems like it may be an oslo messaging bug because in order for us to catch and ignore this in Neutron we would need to import AMQP directly for the exception. Directly depending on AMQP imports breaks the olso.messaging abstraction.

From http://logs.openstack.org/85/480185/4/check/gate-grenade-dsvm-ironic-inspector-ubuntu-xenial/6ef1e75/logs/new/screen-q-svc.txt.gz?level=ERROR#_2017-07-19_12_53_37_858 :

2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation [req-8d272d85-8fc0-4aeb-9180-2bf92cfc67cc tempest-NetworksTest-1317585658 tempest-NetworksTest-1317585658] DELETE failed.: NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation Traceback (most recent call last):
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 683, in __call__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.invoke_controller(controller, args, kwargs, state)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/pecan/core.py", line 574, in invoke_controller
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation result = controller(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 70, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/utils.py", line 109, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/pecan_wsgi/controllers/resource.py", line 78, in delete
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return self.plugin_deleter(*deleter_args)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 929, in delete_network
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return super(Ml2Plugin, self).delete_network(context, id)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/db_base_plugin_v2.py", line 442, in delete_network
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.delete_port(context.elevated(), port_id)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/common/utils.py", line 665, in inner
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(self, context, *args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 164, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return method(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 94, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation setattr(e, '_RETRY_EXCEEDED', True)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 90, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 150, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ectxt.value = e.inner_exc
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 129, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation LOG.debug("Retry wrapper got retriable exception: %s", e)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.force_reraise()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation six.reraise(self.type_, self.value, self.tb)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/db/api.py", line 125, in wrapped
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return f(*dup_args, **dup_kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1479, in delete_port
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation context, port, router_ids, bound_mech_contexts)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/plugin.py", line 1501, in _post_delete_port
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.notifier.port_delete(context, port['id'])
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/opt/stack/new/neutron/neutron/plugins/ml2/rpc.py", line 412, in port_delete
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation cctxt.cast(context, 'port_delete', port_id=port_id)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 147, in cast
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.transport._send(self.target, msg_ctxt, msg, retry=self.retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation timeout=timeout, retry=retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 578, in send
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation retry=retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 552, in _send
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation conn.fanout_send(target.topic, msg, retry=retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1286, in fanout_send
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self._ensure_publishing(self._publish, exchange, msg, retry=retry)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1160, in _ensure_publishing
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.ensure(method, retry=retry, error_callback=_error_callback)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 807, in ensure
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation ret, channel = autoretry_method()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 494, in _ensured
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return fun(*args, **kwargs)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 570, in __call__
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return fun(*args, channel=channels[0], **kwargs), channels[0]
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 796, in execute_method
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation method()
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1192, in _publish
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation compression=self.kombu_compression)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 181, in publish
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation exchange_name, declare,
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 203, in _publish
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation mandatory=mandatory, immediate=immediate,
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 1759, in basic_publish_confirm
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.wait(spec.Basic.Ack)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 93, in wait
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation self.connection.drain_events(timeout=timeout)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 485, in drain_events
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation while not self.blocking_read(timeout):
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 491, in blocking_read
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation return self.on_inbound_frame(frame)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 69, in on_frame
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation callback(channel, method_sig, buf, None)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 495, in on_inbound_method
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation method_sig, payload, content,
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 142, in dispatch_method
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation listener(*args)
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 293, in _on_close
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation reply_code, reply_text, (class_id, method_id), ChannelError,
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation NotFound: Basic.publish: (404) NOT_FOUND - no exchange 'q-agent-notifier-port-delete_fanout' in vhost '/'
2017-07-19 12:53:37.858 6397 ERROR neutron.pecan_wsgi.hooks.translation

Tags:

Kevin Benton (kevinbenton) on 2017-07-19

Changed in neutron:
status:	New → Confirmed
importance:	Undecided → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-19: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/485365

Changed in neutron:
assignee:	nobody → Kevin Benton (kevinbenton)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-24: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/485365
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=93d45fe1ab896c1ec9d183d9ab3c4248cf0aa5ea
Submitter: Jenkins
Branch: master

commit 93d45fe1ab896c1ec9d183d9ab3c4248cf0aa5ea
Author: Kevin Benton <email address hidden>
Date: Wed Jul 19 15:00:06 2017 -0700

Ignore cast exceptions in AgentNotifierApi

    After an upgrade we can get NotFound exceptions from AMQP after there
    are no longer any agents interested in listening to that exchange.
    These should be ignored and not propagated up since they don't indicate
    any kind of issue.

    However, oslo messaging is not translating the exceptions into an
    oslo.messaging exception so there is no way to catch them without
    importing amqp directly.

    As a workround to unblock Ironic, this patch just proposes a generic
    catch of all exceptions during a cast. These casts occur after work
    has been done so raising them back to the user doesn't do anything
    useful anyway.

Once we have an oslo.messaging exception to catch we can narrow it to
that.

Change-Id: I15cc2d6ae48e505c2da121880e27481dedf36d3b
Partial-Bug: #1705351

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-24: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/486687

Changed in neutron:
assignee:	Kevin Benton (kevinbenton) → Ihar Hrachyshka (ihar-hrachyshka)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-07-24: Fix proposed to oslo.messaging (master)

Fix proposed to branch: master
Review: https://review.openstack.org/486706

Changed in oslo.messaging:
assignee:	nobody → Ken Giusti (kgiusti)
status:	New → In Progress

Revision history for this message

Ihar Hrachyshka (ihar-hrachyshka) wrote on 2017-08-01:

Since the immediate issue is fixed with a neutron patch, I think we should close it (for neutron). I leave it up to Kevin to decide though.

Changed in neutron:
assignee:	Ihar Hrachyshka (ihar-hrachyshka) → Kevin Benton (kevinbenton)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-02: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/486687
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=85d8f6f1c8c51f32b6ec2f6fdc8cc09bbcc060eb
Submitter: Jenkins
Branch: master

commit 85d8f6f1c8c51f32b6ec2f6fdc8cc09bbcc060eb
Author: Ihar Hrachyshka <email address hidden>
Date: Mon Jul 24 09:22:58 2017 -0700

Catch exceptions for all rpc casts

    I15cc2d6ae48e505c2da121880e27481dedf36d3b catches exceptions for
    specific RPC endpoints affected by a recent change related to
    push-notifications. There may be more changes like that in the future,
    so instead of fixing them one by one, this patch consistently catches
    exceptions from all cast calls for all RPC clients.

Change-Id: Ia7e6cd717758a9d5b18fe9cb07c55938f52040ce
Partial-Bug: #1705351

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-02: Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/490139

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-03: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/490305

Ihar Hrachyshka (ihar-hrachyshka) on 2017-08-03

Changed in neutron:
milestone:	none → pike-rc1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-04: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/490305
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0b5750ca518314a10266f25cb09997b7a859ab1e
Submitter: Jenkins
Branch: master

commit 0b5750ca518314a10266f25cb09997b7a859ab1e
Author: Kevin Benton <email address hidden>
Date: Wed Aug 2 20:14:10 2017 -0700

Sinkhole workaround for old topics

    This patch introduces a workaround on the Neutron side
    for bug 1705351 that results in the Neutron server blocking
    trying to send to topics the agents no longer subscribe to.

    The workaround is to just subscribe to those topics and do
    nothing with the messages that come in until oslo.messaging
    can properly recover from loss of listeners.

Change-Id: I946a33dfd0c0da26bb47b524f75f53bf59d3fbd5
Closes-Bug: #1705351

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-08: Related fix merged to neutron (master)

#10

Reviewed: https://review.openstack.org/490139
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0b9748e1f9788d1ad7b157fe9eb16666a00da420
Submitter: Jenkins
Branch: master

commit 0b9748e1f9788d1ad7b157fe9eb16666a00da420
Author: Ihar Hrachyshka <email address hidden>
Date: Wed Aug 2 11:37:06 2017 -0700

Degrade log message for missing fanout exchange to debug

    In upgrade scenario, this is a regular case (no agents created the
    exchange, so fanout is ineffective), so logging it using ERROR level is
    misleading and against guidelines.

Change-Id: If098071c9c19490af7eb0863dbab83e488019bd7
Related-Bug: #1705351

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-09: Related fix proposed to neutron (master)

#11

Related fix proposed to branch: master
Review: https://review.openstack.org/492165

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-11: Fix included in openstack/neutron 11.0.0.0rc1

#12

This issue was fixed in the openstack/neutron 11.0.0.0rc1 release candidate.

Ihar Hrachyshka (ihar-hrachyshka) on 2017-08-14

tags:	added: neutron-proactive-backport-potential
tags:	added: neutron-easy-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-16: Change abandoned on neutron (master)

#13

Change abandoned by Ken Giusti (<email address hidden>) on branch: master
Review: https://review.openstack.org/492165
Reason: Kevin's patch solves this.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-08-22: Fix merged to oslo.messaging (master)

#14

Reviewed: https://review.openstack.org/486706
Committed: https://git.openstack.org/cgit/openstack/oslo.messaging/commit/?id=f059bba6ae9642675bc1c9004a4a4adc5d1b19a6
Submitter: Jenkins
Branch: master

commit f059bba6ae9642675bc1c9004a4a4adc5d1b19a6
Author: Kenneth Giusti <email address hidden>
Date: Mon Jul 24 13:36:17 2017 -0400

Prevent rabbit from raising unexpected exceptions

    Publishing a message using the kombu connection autoretry method may
    allow exceptions from the py-amqp library to be raised up to the
    application. This does not conform to the documented oslo.messaging
    API.

Enhance the try except block to capture any exception and translate it
into a MessageDeliveryFailure.

    There are a few cases where exceptions will be raised during autoretry
    publishing: recoverable connection or channel errors, and
    non-recoverable connection or channel errors.

autoretry will only retry if the error is recoverable. Non recoverable
errors are re-raised immediately regardless of the retry count.

    In the case of a recoverable error it seems unlikely that retrying
    either the connection or the channel yet again is going to get us
    anywhere, so in this case we simply clean up the channel state, log an
    error and fail the operation.

    In the case of non-recoverable error we are out of luck (think
    authentication failure) - further retrying will not help. Best we can
    do is clean up state and log the heck out of it.

    Change-Id: I2f65d2ee19a8c3e9a323b30404abbf0cbb45a216
    Closes-Bug: #1705351
    Closes-Bug: #1707160

Changed in oslo.messaging:
status:	In Progress → Fix Released

Revision history for this message

Jakub Libosvar (libosvar) wrote on 2017-08-25:

#15

I understand it that this issue existed only between Pike and Ocata, hence backports are not needed here.

tags:

removed: neutron-easy-proactive-backport-potential neutron-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-09-06: Fix included in openstack/oslo.messaging 5.31.0

#16

This issue was fixed in the openstack/oslo.messaging 5.31.0 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-01-26: Related fix proposed to neutron (master)

#17

Related fix proposed to branch: master
Review: https://review.openstack.org/538318

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-02-15: Related fix merged to neutron (master)

#18

Reviewed: https://review.openstack.org/538318
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=46cbe459ea812ec6ca18b2f597f295e0755927ae
Submitter: Zuul
Branch: master

commit 46cbe459ea812ec6ca18b2f597f295e0755927ae
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Jan 26 10:15:03 2018 -0800

ovs: removed sinkhole RPC target for old topics

The oslo.messaging bug that triggered adding the hack is fixed in
Queens, so we don't need it anymore.

Change-Id: I775b466dce19c4168985b19e1ddb938118f48dcb
Related-Bug: #1705351

Bernard Cafarelli (bcafarel) on 2018-05-31

tags:

added: neutron-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-06-15: Related fix proposed to neutron (stable/queens)

#19

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/575688

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-07-03: Related fix merged to neutron (stable/queens)

#20

Reviewed: https://review.openstack.org/575688
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8cf39054ef2ffa5a28c53e5fd39647024b8cc09a
Submitter: Zuul
Branch: stable/queens

commit 8cf39054ef2ffa5a28c53e5fd39647024b8cc09a
Author: Ihar Hrachyshka <email address hidden>
Date: Fri Jan 26 10:15:03 2018 -0800

ovs: removed sinkhole RPC target for old topics

The oslo.messaging bug that triggered adding the hack is fixed in
Queens, so we don't need it anymore.

    Change-Id: I775b466dce19c4168985b19e1ddb938118f48dcb
    Related-Bug: #1705351
    (cherry picked from commit 46cbe459ea812ec6ca18b2f597f295e0755927ae)

tags:

added: in-stable-queens

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1696690

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.