nova-api is not alway killed with clean.sh and unstack.sh

[Adalberto Medeiros]

nova-api process (and sometimes other nova process) keeps running after unstack or clean, even thought he screen session has been killed.

This presents an issue running stack.sh again, where n-api processes are not restarted.

The idea is to guarantee the nova processes are really killed when running clean or unstack.

[Jay Dobies (jdob)]

As mentioned in the description, it's not just nova-api processes. Below is a ps output after running unstack:

ps -ef | grep nova
jdob 3291 1 0 15:34 ? 00:00:46 /usr/bin/python /usr/bin/nova-api
jdob 3746 1 0 15:34 ? 00:00:21 /usr/bin/python /usr/bin/nova-conductor --config-file /etc/nova/nova.conf
jdob 4034 1 0 15:34 ? 00:00:04 /usr/bin/python /usr/bin/nova-network --config-file /etc/nova/nova.conf
jdob 4148 1 0 15:34 ? 00:00:04 /usr/bin/python /usr/bin/nova-scheduler --config-file /etc/nova/nova.conf
jdob 4535 1 0 15:34 ? 00:00:04 /usr/bin/python /usr/bin/nova-consoleauth --config-file /etc/nova/nova.conf
jdob 12074 3291 0 16:59 ? 00:00:00 /usr/bin/python /usr/bin/nova-api
jdob 12075 3291 0 16:59 ? 00:00:00 /usr/bin/python /usr/bin/nova-api
jdob 12078 3291 0 16:59 ? 00:00:00 /usr/bin/python /usr/bin/nova-api
jdob 12079 3291 0 16:59 ? 00:00:00 /usr/bin/python /usr/bin/nova-api
jdob 12080 3291 0 16:59 ? 00:00:00 /usr/bin/python /usr/bin/nova-api
jdob 12081 3291 0 16:59 ? 00:00:00 /usr/bin/python /usr/bin/nova-api
jdob 12082 3291 0 16:59 ? 00:00:00 /usr/bin/python /usr/bin/nova-api
jdob 12083 3291 0 16:59 ? 00:00:00 /usr/bin/python /usr/bin/nova-api
jdob 13801 1 1 17:04 ? 00:00:01 /usr/bin/python /usr/bin/nova-cert --config-file /etc/nova/nova.conf
jdob 13828 1 1 17:04 ? 00:00:01 /usr/bin/python /usr/bin/nova-conductor --config-file /etc/nova/nova.conf
jdob 13854 1 1 17:04 ? 00:00:01 /usr/bin/python /usr/bin/nova-network --config-file /etc/nova/nova.conf
jdob 13870 1 1 17:04 ? 00:00:01 /usr/bin/python /usr/bin/nova-scheduler --config-file /etc/nova/nova.conf
jdob 13887 1 1 17:04 ? 00:00:01 /usr/bin/python /usr/bin/nova-consoleauth --config-file /etc/nova/nova.conf
jdob 14617 3291 8 17:05 ? 00:00:03 /usr/bin/python /usr/bin/nova-api
jdob 14661 3291 9 17:05 ? 00:00:01 /usr/bin/python /usr/bin/nova-api
jdob 14667 3291 7 17:05 ? 00:00:00 /usr/bin/python /usr/bin/nova-api
jdob 14675 3291 10 17:05 ? 00:00:00 /usr/bin/python /usr/bin/nova-api

[Dean Troyer (dtroyer)]

There are situations where service processes appear to be inherited by init and not killed when screen is killed. The circumstances surrounding this are unclear. https://bugs.launchpad.net/devstack/+bug/1183449 is one approach to kill the screen processes indivudually.

[Maru Newby (maru)]

I have the same problem, and it only seems to afflict nova-* processes.

Please consider following some advice from fungi (via #openstack-infra) to avoid future regressions:

"If you're interested in a clean unstack, maybe add a step in devstack-gate scripts after tempest/grenade runs to perform the unstack and then use something like grenade's check-sanity step to confirm the worrysome processes are no longer running."