delete router from horizon causes critical error in neutron logs

[Mike Manuthu]

While deleting a router from the horizon interface the following error is observed in the neutron-agent-container.

root@infra1-neutron-agents-container-efc7805b:~# tail -45 /var/log/neutron/neutron.log
2017-10-23 17:12:15.472 20348 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', '/openstack/venvs/neutron-16.0.1/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-9', '20349'] create_process /openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/agent/linux/utils.py:92
2017-10-23 17:12:15.488 20348 CRITICAL neutron [-] Unhandled error: AssertionError: do not call blocking functions from the mainloop
2017-10-23 17:12:15.488 20348 ERROR neutron Traceback (most recent call last):
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/bin/neutron-keepalived-state-change", line 11, in <module>
2017-10-23 17:12:15.488 20348 ERROR neutron sys.exit(main())
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/cmd/keepalived_state_change.py", line 19, in main
2017-10-23 17:12:15.488 20348 ERROR neutron keepalived_state_change.main()
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/agent/l3/keepalived_state_change.py", line 156, in main
2017-10-23 17:12:15.488 20348 ERROR neutron cfg.CONF.monitor_cidr).start()
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/agent/linux/daemon.py", line 253, in start
2017-10-23 17:12:15.488 20348 ERROR neutron self.run()
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/agent/l3/keepalived_state_change.py", line 69, in run
2017-10-23 17:12:15.488 20348 ERROR neutron for iterable in self.monitor:
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/agent/linux/async_process.py", line 261, in _iter_queue
2017-10-23 17:12:15.488 20348 ERROR neutron yield queue.get(block=block)
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/eventlet/queue.py", line 313, in get
2017-10-23 17:12:15.488 20348 ERROR neutron return waiter.wait()
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/eventlet/queue.py", line 141, in wait
2017-10-23 17:12:15.488 20348 ERROR neutron return get_hub().switch()
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
2017-10-23 17:12:15.488 20348 ERROR neutron return self.greenlet.switch()
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
2017-10-23 17:12:15.488 20348 ERROR neutron self.wait(sleep_time)
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait
2017-10-23 17:12:15.488 20348 ERROR neutron presult = self.do_poll(seconds)
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll
2017-10-23 17:12:15.488 20348 ERROR neutron return self.poll.poll(seconds)
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/agent/l3/keepalived_state_change.py", line 133, in handle_sigterm
2017-10-23 17:12:15.488 20348 ERROR neutron self._kill_monitor()
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/agent/l3/keepalived_state_change.py", line 130, in _kill_monitor
2017-10-23 17:12:15.488 20348 ERROR neutron run_as_root=True)
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 223, in kill_process
2017-10-23 17:12:15.488 20348 ERROR neutron execute(['kill', '-%d' % signal, pid], run_as_root=run_as_root)
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 131, in execute
2017-10-23 17:12:15.488 20348 ERROR neutron _stdout, _stderr = obj.communicate(_process_input)
2017-10-23 17:12:15.488 20348 ERROR neutron File "/usr/lib/python2.7/subprocess.py", line 800, in communicate
2017-10-23 17:12:15.488 20348 ERROR neutron return self._communicate(input)
2017-10-23 17:12:15.488 20348 ERROR neutron File "/usr/lib/python2.7/subprocess.py", line 1419, in _communicate
2017-10-23 17:12:15.488 20348 ERROR neutron stdout, stderr = self._communicate_with_select(input)
2017-10-23 17:12:15.488 20348 ERROR neutron File "/usr/lib/python2.7/subprocess.py", line 1520, in _communicate_with_select
2017-10-23 17:12:15.488 20348 ERROR neutron rlist, wlist, xlist = select.select(read_set, write_set, [])
2017-10-23 17:12:15.488 20348 ERROR neutron File "/openstack/venvs/neutron-16.0.1/lib/python2.7/site-packages/eventlet/green/select.py", line 40, in select
2017-10-23 17:12:15.488 20348 ERROR neutron assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
2017-10-23 17:12:15.488 20348 ERROR neutron AssertionError: do not call blocking functions from the mainloop
2017-10-23 17:12:15.488 20348 ERROR neutron

Please note that the router is successfully deleted.

Openstack release via https://docs.openstack.org/openstack-ansible/pike/

(neutron-16.0.1) root@infra1-neutron-server-container-5cb0c95d:~# pip freeze | grep neutron
neutron==11.0.1.dev46
neutron-lib==1.9.1
python-neutronclient==6.5.0

[James Anziano (janzian)]

I can confirm I also get errors trying this on the same versions, though my errors are different [1]. On upstream however, this is not reproducible, so it appears to be fixed going forward.

[1] http://paste.openstack.org/show/624635/

[James Anziano (janzian)]

Ignore my paste, realized afterwards what those were :)

[Mike Manuthu (mikemanuthu)]

Good to know.

cheers,