UPDATE:
to reproduce the bug:
log into the neutron_api container :
docker exec -it --user root neutron_api bash
ps fax| grep neutron_api
()[root@undercloud /]# ps fax
PID TTY STAT TIME COMMAND
115 ? Ss 0:00 bash
140 ? S+ 0:00 \_ top
44 ? Ss 0:00 bash
241 ? R+ 0:00 \_ ps fax
1 ? Ss 0:00 /usr/local/bin/dumb-init /bin/bash /usr/local/bin/kolla_sta
7 ? Ss 0:05 /usr/bin/python2 /usr/bin/neutron-server --config-file /usr
27 ? S 0:08 \_ /usr/bin/python2 /usr/bin/neutron-server --config-file
28 ? S 0:00 \_ /usr/bin/python2 /usr/bin/neutron-server --config-file
29 ? S 0:03 \_ /usr/bin/python2 /usr/bin/neutron-server --config-file
30 ? S 0:03 \_ /usr/bin/python2 /usr/bin/neutron-server --config-file
31 ? S 0:03 \_ /usr/bin/python2 /usr/bin/neutron-server --config-file
32 ? R 5:27 \_ /usr/bin/python2 /usr/bin/neutron-server --config-file
Kill the last on pid ( 32 ) with sigup-
kill -1 32
check the server.log after few seconds :
2018-12-26 00:00:36.077 40997 ERROR oslo_service.service [-] Error starting thread.: RuntimeError: A fixed interval looping call can only run one function at a time
in our environment, this occurs without someone issuing kill -1 , but just after 4 days more or less there's a sigup and docker becomes unhealthy.
UPDATE:
to reproduce the bug:
log into the neutron_api container :
docker exec -it --user root neutron_api bash bin/dumb- init /bin/bash /usr/local/ bin/kolla_ sta neutron- server --config-file /usr neutron- server --config-file neutron- server --config-file neutron- server --config-file neutron- server --config-file neutron- server --config-file neutron- server --config-file
ps fax| grep neutron_api
()[root@undercloud /]# ps fax
PID TTY STAT TIME COMMAND
115 ? Ss 0:00 bash
140 ? S+ 0:00 \_ top
44 ? Ss 0:00 bash
241 ? R+ 0:00 \_ ps fax
1 ? Ss 0:00 /usr/local/
7 ? Ss 0:05 /usr/bin/python2 /usr/bin/
27 ? S 0:08 \_ /usr/bin/python2 /usr/bin/
28 ? S 0:00 \_ /usr/bin/python2 /usr/bin/
29 ? S 0:03 \_ /usr/bin/python2 /usr/bin/
30 ? S 0:03 \_ /usr/bin/python2 /usr/bin/
31 ? S 0:03 \_ /usr/bin/python2 /usr/bin/
32 ? R 5:27 \_ /usr/bin/python2 /usr/bin/
Kill the last on pid ( 32 ) with sigup-
kill -1 32
check the server.log after few seconds : service [-] Error starting thread.: RuntimeError: A fixed interval looping call can only run one function at a time
2018-12-26 00:00:36.077 40997 ERROR oslo_service.
in our environment, this occurs without someone issuing kill -1 , but just after 4 days more or less there's a sigup and docker becomes unhealthy.