Fullstack linux bridge agent sometimes refuses to die during test clean up, failing the test

[Assaf Muller]

Paste of failure:
http://paste.openstack.org/show/491014/

When looking at the LB agent logs, you start seeing RPC errors as neutron-server is unable to access the DB. What's happening is that fullstack times out trying to kill the LB agent and moves on to other clean ups. It deletes the DB for the test, but the agents and neutron-server live on, resulting in errors trying to access the DB. The DB errors are essentially unrelated - The root cause is that the agent refuses to die for an unknown reason.

The code that tries to stop the agent is AsyncProcess.stop(block=True, signal=9).
Another detail that might be relevant is that the agent lives in a namespace.

To reproduce locally, go to the VM running the fullstack tests and load all CPUs to 100%, then run:
tox -e dsvm-fullstack TestLinuxBridgeConnectivitySameNetwork

[Assaf Muller (amuller)]

We had the same symptom with the OVS agent, that was fixed via https://review.openstack.org/#/c/234770/. I have absolutely no idea if it's related at this point.

[Assaf Muller (amuller)]

I started looking in to this. When trying to kill the Linuxbridge agent, we use AsyncProcess.stop, which calls _kill_process. It looks like for failed runs, it's trying to kill the wrong pid. I added a print statement, and the pid it's trying to kill is not there, while the linuxbridge agent is alive and well with a different pid. We get a Runtime error saying that the process was not found, then utils.wait_until_true(lambda: not self.is_active()) times out as it does wait on the correct pid.

[John Schwarz (jschwarz)]

Interesting that the pid doesn't exist. Out of the top of my head, AFAIR, async_process has something that can trace child processes, etc. Perhaps the LinuxBridge agent is daemonizing, killing the parent process in the progress and preventing async_process to work properly?

[Assaf Muller (amuller)]

@John, keep in mind that this sometimes works but sometimes doesn't.

[Assaf Muller (amuller)]

Under normal operation, pstree returns 'sudo ip netns...' > 'linuxbridge agent'. Typically get_root_helper_child_pid returns the child pid (linuxbridge agent), we kill that pid, and all is well. I added a bunch of prints and found out that when it fails, get_root_helper_child_pid returns a child of linuxbridge_agent (??), 'ps -p %s' output for it doesn't find anything, we try to kill it (Which obviously fails) and the agent lives on. Currently not sure what child is get_root_helper_child_pid finding. Could it be just any process that the LB agent happens to have executed when we checked for its children, like 'ip' / 'brctl' commands and so on?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/294798

[Assaf Muller (amuller)]

I pushed https://review.openstack.org/#/c/294798/ , I can no longer reproduce this bug in my local environment.

[John Schwarz (jschwarz)]

@Assaf, it isn't clear from your comment - can you no longer reproduce with the specified patch (hinting it might fix it), or even without the specified patch (something else changed)?

Also, I added Kuba as a subscriber - IIRC he worked on the whole get_root_helper_child_pid (or something around that neighborhood) and might be able to provide a "why this was implemented like this". I'm thinking about if that function always return the child pid, this should popped up a lot sooner (ie. ovs agent with ip monitor, l3 agent with keepalived...)

Perhaps the cause isn't get_root_helper_child_pid, but the way linuxbridge spawn new processes?

[Assaf Muller (amuller)]

I mean that the patch fixes the bug.

[John Schwarz (jschwarz)]

A deeper look at 'ps -ef' shows that most (if not all) processes spawned by other agents (l3 agent, ovs agent...) are daemonized (keepalived, ip monitor... have their PPID == 1), so that explains why get_root_helper_child_pid doesn't see them and why it didn't reproduce sooner. I'm fine with modifying the function as Assaf proposed.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/299920

[OpenStack Infra (hudson-openstack)]

Change abandoned by Jakub Libosvar (<email address hidden>) on branch: master
Review: https://review.openstack.org/299920
Reason: Its purpose has been fulfilled.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/294798
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=fd93e19f2a415b3803700fc491749daba01a4390
Submitter: Jenkins
Branch: master

commit fd93e19f2a415b3803700fc491749daba01a4390
Author: Assaf Muller <email address hidden>
Date: Fri Mar 18 16:29:26 2016 -0400

    Change get_root_helper_child_pid to stop when it finds cmd

    get_root_helper_child_pid recursively finds the child of pid,
    until it can no longer find a child. However, the intention is
    not to find the deepest child, but to strip away root helpers.
    For example 'sudo neutron-rootwrap x' is supposed to find the
    pid of x. However, in cases 'x' spawned quick lived children of
    its own (For example: ip / brctl / ovs invocations),
    get_root_helper_child_pid returned those pids if called in
    the wrong time.

    Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e
    Closes-Bug: #1558819

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/321768

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/321770

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/neutron 9.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/321770
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d7c8e53d9008ae9a9338caea84634e1ad57bed0c
Submitter: Jenkins
Branch: stable/liberty

commit d7c8e53d9008ae9a9338caea84634e1ad57bed0c
Author: Assaf Muller <email address hidden>
Date: Fri Mar 18 16:29:26 2016 -0400

    Change get_root_helper_child_pid to stop when it finds cmd

    get_root_helper_child_pid recursively finds the child of pid,
    until it can no longer find a child. However, the intention is
    not to find the deepest child, but to strip away root helpers.
    For example 'sudo neutron-rootwrap x' is supposed to find the
    pid of x. However, in cases 'x' spawned quick lived children of
    its own (For example: ip / brctl / ovs invocations),
    get_root_helper_child_pid returned those pids if called in
    the wrong time.

    Conflicts:
     neutron/tests/contrib/functional-testing.filters

    Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e
    Closes-Bug: #1558819
    (cherry picked from commit fd93e19f2a415b3803700fc491749daba01a4390)
    (cherry picked from commit 1d714c35add69ba1237ba63a5725e336892f3b9f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/321768
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1d714c35add69ba1237ba63a5725e336892f3b9f
Submitter: Jenkins
Branch: stable/mitaka

commit 1d714c35add69ba1237ba63a5725e336892f3b9f
Author: Assaf Muller <email address hidden>
Date: Fri Mar 18 16:29:26 2016 -0400

    Change get_root_helper_child_pid to stop when it finds cmd

    get_root_helper_child_pid recursively finds the child of pid,
    until it can no longer find a child. However, the intention is
    not to find the deepest child, but to strip away root helpers.
    For example 'sudo neutron-rootwrap x' is supposed to find the
    pid of x. However, in cases 'x' spawned quick lived children of
    its own (For example: ip / brctl / ovs invocations),
    get_root_helper_child_pid returned those pids if called in
    the wrong time.

    Conflicts:
     neutron/tests/contrib/functional-testing.filters

    Change-Id: I582aa5c931c8bfe57f49df6899445698270bb33e
    Closes-Bug: #1558819
    (cherry picked from commit fd93e19f2a415b3803700fc491749daba01a4390)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 7.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 8.3.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 7.2.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 8.3.0 release.