Comment 9 for bug 1680183

We still hit the issue, though the trace is a bit different now:

http://logs.openstack.org/38/284738/69/check/gate-neutron-dsvm-fullstack-ubuntu-xenial/2e022c5/logs/syslog.txt.gz

Apr 28 17:24:20 ubuntu-xenial-rax-ord-8648308 neutron-keepalived-state-change[21615]: 2017-04-28 17:24:20.423 21615 CRITICAL neutron [-] AssertionError: do not call blocking functions from the mainloop
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron Traceback (most recent call last):
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/.tox/dsvm-fullstack/bin/neutron-keepalived-state-change", line 10, in <module>
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron sys.exit(main())
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/neutron/cmd/keepalived_state_change.py", line 19, in main
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron keepalived_state_change.main()
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/neutron/agent/l3/keepalived_state_change.py", line 156, in main
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron cfg.CONF.monitor_cidr).start()
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/neutron/agent/linux/daemon.py", line 253, in start
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron self.run()
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/neutron/agent/l3/keepalived_state_change.py", line 69, in run
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron for iterable in self.monitor:
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/neutron/agent/linux/async_process.py", line 261, in _iter_queue
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron yield queue.get(block=block)
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/queue.py", line 313, in get
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron return waiter.wait()
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/queue.py", line 141, in wait
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron return get_hub().switch()
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron return self.greenlet.switch()
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in run
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron self.wait(sleep_time)
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/poll.py", line 85, in wait
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron presult = self.do_poll(seconds)
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/epolls.py", line 62, in do_poll
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron return self.poll.poll(seconds)
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/neutron/agent/l3/keepalived_state_change.py", line 133, in handle_sigterm
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron self._kill_monitor()
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/neutron/agent/l3/keepalived_state_change.py", line 130, in _kill_monitor
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron run_as_root=True)
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 225, in kill_process
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron execute(['kill', '-%d' % signal, pid], run_as_root=run_as_root)
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/neutron/agent/linux/utils.py", line 131, in execute
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron _stdout, _stderr = obj.communicate(_process_input)
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/usr/lib/python2.7/subprocess.py", line 800, in communicate
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron return self._communicate(input)
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/usr/lib/python2.7/subprocess.py", line 1419, in _communicate
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron stdout, stderr = self._communicate_with_select(input)
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/usr/lib/python2.7/subprocess.py", line 1520, in _communicate_with_select
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron rlist, wlist, xlist = select.select(read_set, write_set, [])
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/green/select.py", line 40, in select
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron assert hub.greenlet is not current, 'do not call blocking functions from the mainloop'
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron AssertionError: do not call blocking functions from the mainloop
                                                                                      2017-04-28 17:24:20.423 21615 ERROR neutron

We probably need to rework how we handle the signal, not calling execute() from inside the handler.