fullstack runs leave neutron-server forked processes running

[Assaf Muller]

Since by default api_workers = Number of CPUs on the machine, the neutron-server forks itself when it starts. Those child processes are left behind on both successful and failed fullstack runs. This is possibly related to bug https://bugs.launchpad.net/neutron/+bug/1487548.

[Armando Migliaccio (armando-migliaccio)]

Can this pose a stability issue to the gate job? Would consecutive runs interfere with one another on a local box?

[Armando Migliaccio (armando-migliaccio)]

Is there a simple check that one can do to confirm the issue? Do you intend to work on it yourself? If not, it would be nice to provide more info so that the triaging would be faster.

Would the execution of:

tox -edsvm-fullstack ; ps -aux | grep neutron

suffice, to demonstrate the leaking?

[Assaf Muller (amuller)]

> Can this pose a stability issue to the gate job?

I don't know yet. The fullstack job is extremely unstable right now, I've not determined why yet.

> Would consecutive runs interfere with one another on a local box?

Absolutely. Given enough runs you eventually run out of DB connections (Since the children neutron-server processes aren't killed, they maintain their DB connections) and all following test runs fail.

> Do you intend to work on it yourself?

Not at this time. I was hoping this bug report would illicit a discussion so that the correct solution could surface.

> If not, it would be nice to provide more info so that the triaging would be faster.

ps aux | grep neutron-server shows an empty list
tox -e dsvm-fullstack <path.to.a.single.test>
ps aux | grep neutron-server shows only children of parent neutron-server, parent was killed

We clean up resources (Including the neutron-server) with a kill -9. That leaves the children behind and they're never cleaned up. The issue is easily reproducible I'm just not sure what is the correct solution.

[Jakub Libosvar (libosvar)]

Isn't this related to the thing we still use SIGKILL to terminate neutron-server? This way orphaned child processes won't get a chance to be terminated by the parent neutron-server. Receiving SIGTERM in neutron-server will stop child processes.

[Jakub Libosvar (libosvar)]

I just read the last paragraph of Assaf's comment and now I feel dumb...I should read first and then write. I apologize.

[Assaf Muller (amuller)]

The other bug wasn't tagged with 'fullstack' so I missed it.