neutron

Neutron OVS agent fails to start when neutron-server is not available

Bug #1694524 reported by Ihar Hrachyshka on 2017-05-30

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	Invalid	High	Kevin Benton	neutron pike-2 "p-2"
	tripleo	Fix Released	Critical	Unassigned	tripleo pike-2 "pike-2"

Bug Description

2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp [req-34115ed3-3043-4fcb-ba3f-ab0e4eb0e83c - - - - -] Agent main thread died of an exception
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp Traceback (most recent call last):
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 40, in agent_main_wrapper
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp ovs_agent.main(bridge_classes)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2166, in main
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 180, in __init__
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp self.setup_rpc()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp return f(*args, **kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 362, in setup_rpc
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp self.plugin_rpc = OVSPluginApi(topics.PLUGIN)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 182, in __init__
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp self.remote_resource_cache = create_cache_for_l2_agent()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 174, in create_cache_for_l2_agent
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp rcache.bulk_flood_cache()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/agent/resource_cache.py", line 55, in bulk_flood_cache
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp for resource in puller.bulk_pull(context, rtype):
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_log/helpers.py", line 48, in wrapper
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp return method(*args, **kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/api/rpc/handlers/resources_rpc.py", line 109, in bulk_pull
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp version=resource_type_cls.VERSION, filter_kwargs=filter_kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 174, in call
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp time.sleep(wait)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp self.force_reraise()
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp six.reraise(self.type_, self.value, self.tb)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 151, in call
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp return self._original_context.call(ctxt, method, **kwargs)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp retry=self.retry)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 123, in _send
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp timeout=timeout, retry=retry)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 505, in send
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp retry=retry)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 494, in _send
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp result = self._waiter.wait(msg_id, timeout)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 386, in wait
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp message = self.waiters.get(msg_id, timeout=timeout)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 275, in get
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp 'to message ID %s' % msg_id)
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp MessagingTimeout: Timed out waiting for a reply to message ID c79afd6fe556408bbaca568e599feada
2017-05-30 18:58:01.947 27929 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_ryuapp
2017-05-30 18:58:02.051 27929 INFO oslo_rootwrap.client [-] Stopping rootwrap daemon process with pid=28388

https://logs.rdoproject.org/weirdo-generic-puppet-openstack-scenario001/1754/weirdo-project/logs/neutron/openvswitch-agent.txt.gz

This is probably because of push-notifications work.

Tags:

Ihar Hrachyshka (ihar-hrachyshka) on 2017-05-30

Changed in neutron:
importance:	Undecided → High
tags:	added: ovs
Changed in neutron:
status:	New → Confirmed
milestone:	none → pike-2

Kevin Benton (kevinbenton) on 2017-05-30

Changed in neutron:
assignee:	nobody → Kevin Benton (kevinbenton)

Emilien Macchi (emilienm) on 2017-05-31

Changed in tripleo:
status:	New → Triaged
milestone:	none → pike-2
importance:	Undecided → Critical
tags:	added: alert ci promotion-blocker