Grenade old n-api doesn't stop

[Sylvain Bauza]

We're seeing a logstash pattern occuring regularly http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22AssertionError:%20Cannot%20switch%20to%20MAINLOOP%20from%20MAINLOOP%5C%22&from=7d#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20Cannot%20switch%20to%20MAINLOOP%20from%20MAINLOOP%5C%22%20AND%20project%3A%5C%22openstack%2Fnova%5C%22

old n-api in Grenade is unable to stop. After further digging, it seems that eventlet is raising an AssertionError when oslo.service tries to stop the WSGI stack:

2016-02-09 12:39:06.991 30139 INFO nova.wsgi [-] WSGI server has stopped.
    handler(signo, frame)
  File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 364, in _sigterm
    SignalHandler().clear()
  File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 117, in __call__
    with lockutils.lock('singleton_lock', semaphores=cls._semaphores):
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-02-09 12:39:06.991 30068 INFO oslo_service.service [-] Child 30080 exited with status 0
    return self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 210, in lock
    with int_lock:
  File "/usr/lib/python2.7/threading.py", line 467, in acquire
2016-02-09 12:39:06.994 30068 INFO oslo_service.service [-] Child 30139 exited with status 0
    self.__cond.wait()
  File "/usr/lib/python2.7/threading.py", line 339, in wait
    waiter.acquire()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/semaphore.py", line 113, in acquire
    hubs.get_hub().switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 280, in switch
    assert cur is not self.greenlet, 'Cannot switch to MAINLOOP from MAINLOOP'
AssertionError: Cannot switch to MAINLOOP from MAINLOOP
Logged from file lockutils.py, line 212

[Davanum Srinivas (DIMS) (dims-v)]

Should be fixed in oslo.service 0.9.1:
http://markmail.org/message/hcglxo7s43rbazgr