os.kill(SIGTERM) does not finish and timeouts

[Rodolfo Alonso]

Since [1], the process signals are sent using os.kill() method.

It is happening that, in some cases, the method never returns and raises a timeout.

Because os.kill() is a blocking method, we should find a way to monitor this process and end it properly.

Error logs:
- https://bec7cc51aed17868a822-df0c8eba46b67aa6178498e0cf2208c0.ssl.cf1.rackcdn.com/779310/6/check/neutron-functional-with-uwsgi/e6d8aae/testr_results.html
- https://b19b5ab35f9ced465dd5-2eb50734132c0e56282483bcdf57bf8a.ssl.cf5.rackcdn.com/782587/5/check/neutron-functional-with-uwsgi/8a6a6d3/testr_results.html
- https://52682a1fca01b92ead5c-4c2761265b996c18f02000b5e0f64005.ssl.cf5.rackcdn.com/782275/1/check/neutron-functional-with-uwsgi/b890243/testr_results.html

Snippet: http://paste.openstack.org/show/803873/

[1]https://review.opendev.org/c/openstack/neutron/+/681671

[Lajos Katona (lajos-katona)]

I suppose this is the patch for it: https://review.opendev.org/c/openstack/neutron/+/782842

[Lajos Katona (lajos-katona)]

os.kill related patch was reverted: https://review.opendev.org/c/openstack/neutron/+/782972

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

I'm developing a possible solution in oslo.privsep. [1] implements a privsep decorator with timeout. When the client side timeouts, the method raises a timeout exception. That can be catch by Neutron.

The problem: the commands will continue running in the daemon and nothing will stop the thread running it until the command returns. This design is unstable because we can run out of threads very quick.

A possible alternative could be to execute those commands spawning a new process, instead a new thread. This process can be monitored and, if needed, killed by the daemon if does not reply on time. But this is something that I need to propose in this patch.

Regards.

[1]https://review.opendev.org/c/openstack/oslo.privsep/+/782981

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

The problem is still present even with the os.kill method revert [1].

[1]https://review.opendev.org/c/openstack/neutron/+/782972

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/791243

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/791986

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791986
Committed: https://opendev.org/openstack/neutron/commit/54420d04dd2abc2bdeb78b9490032e1b1ebd3fac
Submitter: "Zuul (22348)"
Branch: master

commit 54420d04dd2abc2bdeb78b9490032e1b1ebd3fac
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue May 18 14:27:17 2021 +0000

    Skip "test_keepalived_spawns_conflicting_*" tests

    SIGHUP signal command to stop those processes does not return and
    the test (and eventually the whole testsuit) timeouts.

    Until a proper fix is implemented, these tests will be skipped
    temporarily.

    Change-Id: I592d8b7bb2afe5cd93cbb0d0ea7062f1b724ff2a
    Related-Bug: #1921154

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/791243
Reason: This is bad idea

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

The CI issue is now solved, lowering the importance of the bug.