Failed to stop nova-api in grenade tests

[Thomas Herve]

Saw this during a grenade run:

2016-01-26 16:12:58.553 22016 ERROR oslo_service.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo_service/service.py", line 143, in clear
2016-01-26 16:12:58.553 22016 ERROR oslo_service.threadgroup for sig in self._signal_handlers:
2016-01-26 16:12:58.553 22016 ERROR oslo_service.threadgroup RuntimeError: dictionary changed size during iteration

(From http://logs.openstack.org/25/272425/1/gate/gate-grenade-dsvm-heat/b32eda2/).

May be due to a change in oslo, but it's in the old process so I'm not sure it ought to use it.

[Thomas Herve (therve)]

https://review.openstack.org/#/c/260386/ looks suspicious.

[Thomas Herve (therve)]

Ah in fact it's possible that it's https://bugs.launchpad.net/oslo.service/+bug/1524907

[Victor Stinner (vstinner)]

I looked at the traceback. I'm unable to understand exactly why and how the bug was triggered, but it really looks like a reentrant call to the _sigterm() method. A reentrant call to _sigterm() means a reentrant call to clear() which can explain a RuntimeError exception. The second call modifies the dictionary whereas the first call is still iterating on it.

If it's a reentrant call, the second call (the reentrant call) is done when the RuntimeError is raised in the first call. So it's not possible to see the reentrant call in the traceback.

I proposed the change https://review.openstack.org/#/c/272718/ which may (or may not) fix this bug.

[Victor Stinner (vstinner)]

To reproduce the bug, you should try to send quickly a signal twice. The SIGTERM is a good candidate since its signal handler calls clear() and clear() modifies the dictionary of signal handlers, so it can explain the bug.

Reproducing the bug would help to understand it and also to test my change.

[Marian Horban (mhorban)]

Why we can not just add lockers in oslo_service.threadgroup to avoid changing dict during iteration?

[Sean Dague (sdague)]

This is definitely a core oslo.service issue with shutting down, this keeps tripping us up.

[Davanum Srinivas (DIMS) (dims-v)]

16 hits in last week for message:"dictionary changed size during iteration" logstash query. Only one of them is a FAILURE though unrelated to this bug.

-- Dims