[Functional tests] Timeout exception in list_namespace_pids

[Slawek Kaplonski]

Example of error:

ft1.3: neutron.tests.functional.agent.linux.test_keepalived.KeepalivedManagerTestCase.test_keepalived_spawns_conflicting_pid_vrrp_subprocesstesttools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/fixture.py", line 125, in cleanUp
    return self._cleanups(raise_errors=raise_first)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/callmany.py", line 89, in __call__
    reraise(error[0], error[1], error[2])
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/testtools/_compat3x.py", line 16, in reraise
    raise exc_obj.with_traceback(exc_tb)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/callmany.py", line 83, in __call__
    cleanup(*args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 598, in destroy
    for pid in ip_lib.list_namespace_pids(self.name):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/ip_lib.py", line 941, in list_namespace_pids
    return privileged.list_ns_pids(namespace)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 245, in _wrap
    return self.channel.remote_call(name, args, kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 194, in remote_call
    result = self.send_recv((Message.CALL.value, name, args, kwargs))
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/oslo_privsep/comm.py", line 171, in send_recv
    reply = future.result()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/oslo_privsep/comm.py", line 108, in result
    self.condvar.wait()
  File "/usr/lib/python3.6/threading.py", line 295, in wait
    waiter.acquire()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/semaphore.py", line 115, in acquire
    hubs.get_hub().switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 298, in switch
    return self.greenlet.switch()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/hub.py", line 350, in run
    self.wait(sleep_time)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/poll.py", line 80, in wait
    presult = self.do_poll(seconds)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
    return self.poll.poll(seconds)
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-functional/lib/python3.6/site-packages/fixtures/_fixtures/timeout.py", line 52, in signal_handler
    raise TimeoutException()
fixtures._fixtures.timeout.TimeoutException

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22line%20941%2C%20in%20list_namespace_pids%5C%22

[Slawek Kaplonski (slaweq)]

Another example of similar failure: https://e09fc8988ef3a7f5d12b-01a80a447b6e4e9a4734ae7a7e6b5d1e.ssl.cf2.rackcdn.com/681258/6/check/neutron-functional/42208ae/testr_results.html.gz

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

I would like to document here a possible problem when using futures and gevent: https://stackoverflow.com/questions/21104177/using-concurrent-futures-future-with-greenlets-gevent?rq=1. This problem could be difficult to debug.

Another problem we have here is documented in https://bugs.launchpad.net/neutron/+bug/1833721. Now we are going to drop Py2 support, we should invert the calling order for the ip_lib methods: https://review.opendev.org/#/c/666853/

I'll propose again this patch, adding a note saying that this patch breaks the compatibility with Py2.

Regards.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Actually this is something already done: https://review.opendev.org/#/c/683109/

I still need to investigate this problem that usually hits the namespace methods (list, create, delete).

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

Using the following script [1] and adding debug log messages in the pyroute2 library, I found that the method [2] tends to block the execution, not returning anything and blocking other threads, as seen in several log reports.

A possible solution could be to declare just once this variable in the priv_lib module and the passing it to the create and delete methods.

I'll propose a patch for this.

[1] http://paste.openstack.org/show/787322/
[2] https://github.com/svinota/pyroute2/blob/master/pyroute2/netns/__init__.py#L209

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/698039

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/698039
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=af8a8122407331c70e11b0fe02ad0178fa5181f6
Submitter: Zuul
Branch: master

commit af8a8122407331c70e11b0fe02ad0178fa5181f6
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 9 16:13:10 2019 +0000

    Load the glibc library only once for Pyroute2

    Load the glibc library only once, needed in the Pyroute2 methods to
    create and delete a network namespace.

    Change-Id: I95b7b7008f4788a98ef871c4b7aecea839ff2310
    Closes-Bug: #1854462

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 16.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/721698

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/721698
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a57feaf936e0e5ecadb77cf5e1b7efa2c6908ebe
Submitter: Zuul
Branch: stable/train

commit a57feaf936e0e5ecadb77cf5e1b7efa2c6908ebe
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 9 16:13:10 2019 +0000

    Load the glibc library only once for Pyroute2

    Load the glibc library only once, needed in the Pyroute2 methods to
    create and delete a network namespace.

    Change-Id: I95b7b7008f4788a98ef871c4b7aecea839ff2310
    Closes-Bug: #1854462
    (cherry picked from commit af8a8122407331c70e11b0fe02ad0178fa5181f6)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/751209

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/751214

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/751216

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/751209
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e42c15de3b88d135cce4d4684ecd42bcc816065a
Submitter: Zuul
Branch: stable/stein

commit e42c15de3b88d135cce4d4684ecd42bcc816065a
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 9 16:13:10 2019 +0000

    Load the glibc library only once for Pyroute2

    Load the glibc library only once, needed in the Pyroute2 methods to
    create and delete a network namespace.

    Change-Id: I95b7b7008f4788a98ef871c4b7aecea839ff2310
    Closes-Bug: #1854462
    (cherry picked from commit af8a8122407331c70e11b0fe02ad0178fa5181f6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/751214
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b2af92298736be447558d27c2562cd0d001995e3
Submitter: Zuul
Branch: stable/rocky

commit b2af92298736be447558d27c2562cd0d001995e3
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 9 16:13:10 2019 +0000

    Load the glibc library only once for Pyroute2

    Load the glibc library only once, needed in the Pyroute2 methods to
    create and delete a network namespace.

    Conflicts:
        neutron/privileged/agent/linux/ip_lib.py

    Change-Id: I95b7b7008f4788a98ef871c4b7aecea839ff2310
    Closes-Bug: #1854462
    (cherry picked from commit af8a8122407331c70e11b0fe02ad0178fa5181f6)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/751216
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0e14524c6b9d4b8d4c7d434b9fe871f3350f4657
Submitter: Zuul
Branch: stable/queens

commit 0e14524c6b9d4b8d4c7d434b9fe871f3350f4657
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Dec 9 16:13:10 2019 +0000

    Load the glibc library only once for Pyroute2

    Load the glibc library only once, needed in the Pyroute2 methods to
    create and delete a network namespace.

    Conflicts:
        neutron/privileged/agent/linux/ip_lib.py

    Change-Id: I95b7b7008f4788a98ef871c4b7aecea839ff2310
    Closes-Bug: #1854462
    (cherry picked from commit af8a8122407331c70e11b0fe02ad0178fa5181f6)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron queens-eol release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron rocky-eol release.