neutron-openvswitch-agent - error on startup - uncaught exception causes exit of eventloop

[James Page]

I've seen this a few times during setup of the n-ovs-agent process on network nodes (n-gateway) in juju deployments on OpenStack Newton B3+ on Xenial:

2016-09-12 14:41:45.565 20629 WARNING stevedore.named [req-3762dc49-86d7-406b-81fb-5b068416abf9 - - - - -] Could not load neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
2016-09-12 14:41:56.447 20629 ERROR ryu.lib.hub [req-5c9cbccf-7bcb-4c1f-a954-69202bfbcd56 - - - - -] hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ryu/lib/hub.py", line 52, in _launch
    func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 35, in agent_main_wrapper
    ovs_agent.main(bridge_classes)
  File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2175, in main
    agent.daemon_loop()
  File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 154, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2094, in daemon_loop
    self.ovsdb_monitor_respawn_interval) as pm:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/polling.py", line 35, in get_polling_manager
    pm.start()
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/polling.py", line 57, in start
    self._monitor.start(block=True)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ovsdb_monitor.py", line 117, in start
    while not self.is_active():
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line 100, in is_active
    self.pid, self.cmd_without_namespace)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py", line 159, in pid
    run_as_root=self.run_as_root)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 234, in get_root_helper_child_pid
    pid = find_child_pids(pid)[0]
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 167, in find_child_pids
    log_fail_as_error=False)
  File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 118, in execute
    _stdout, _stderr = obj.communicate(_process_input)
  File "/usr/lib/python2.7/subprocess.py", line 800, in communicate
    return self._communicate(input)
  File "/usr/lib/python2.7/subprocess.py", line 1437, in _communicate
    self.wait()
  File "/usr/lib/python2.7/dist-packages/eventlet/green/subprocess.py", line 87, in wait
    eventlet.sleep(check_interval)
  File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 34, in sleep
    hub.switch()
  File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
    return self.greenlet.switch()
Timeout: 5 seconds

Something is spinning - not sure what; the agent is non-responsive and needs a restart to get things working.

[James Page (james-page)]

Something todo with the initial spawn of the ovsdb-monitor process:

2016-09-12 20:18:45.506 20299 DEBUG neutron.agent.linux.async_process [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Launching async process [ovsdb-client monitor Interface name,ofport,external_ids --format=json]. start /usr/lib/python2.7/dist-packages/neutron/agent/linux/async_process.py:109
2016-09-12 20:18:45.508 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Running command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovsdb-client', 'monitor', 'Interface', 'name,ofport,external_ids', '--format=json'] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:83
2016-09-12 20:18:45.602 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Running command: ['ps', '--ppid', '20527', '-o', 'pid='] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:83
2016-09-12 20:18:45.922 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Exit code: 0 execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:140
2016-09-12 20:18:45.935 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Running command: ['ps', '--ppid', '20529', '-o', 'pid='] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:83
2016-09-12 20:18:46.247 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Running command: ['ps', '--ppid', '20527', '-o', 'pid='] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:83
2016-09-12 20:18:46.558 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Exit code: 0 execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:140
2016-09-12 20:18:46.563 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Running command: ['ps', '--ppid', '20529', '-o', 'pid='] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:83
2016-09-12 20:18:46.835 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Running command: ['ps', '--ppid', '20527', '-o', 'pid='] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:83
2016-09-12 20:18:47.183 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Exit code: 0 execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:140
2016-09-12 20:18:47.191 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Running command: ['ps', '--ppid', '20529', '-o', 'pid='] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:83
2016-09-12 20:18:47.539 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465fa460dbc - - - - -] Running command: ['ps', '--ppid', '20527', '-o', 'pid='] create_process /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:83
2016-09-12 20:18:47.842 20299 DEBUG neutron.agent.linux.utils [req-5d68e705-ef4a-449b-8150-e465f...

[James Page (james-page)]

We're running an older ryu version than upstream test with:

  https://github.com/osrg/ryu/commit/f1f0ca2d1688d25924854a635becdcbfef95a5cd

looks likely - I'd like to get ryu 4.5 into yakkety to see if that helps.

[James Page (james-page)]

I guess - 'hub: uncaught exception: Traceback'

[Chris MacNaughton (chris.macnaughton)]

As there have been several releases since this, I'm marking this fix-released against both ryu and neutron. If this is still occurring, please re-open this bug with new version information.