Many fullstack tests failing because of some error in L3 agent

[Slawek Kaplonski]

For example: http://logs.openstack.org/80/536380/2/check/neutron-fullstack/138850e/logs/testr_results.html.gz

Many tests are failing with error:

Traceback (most recent call last):
  File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/fixtures/fixture.py", line 197, in setUp
    self._setUp()
  File "neutron/tests/fullstack/resources/environment.py", line 395, in _setUp
    self.wait_until_env_is_up()
  File "neutron/tests/fullstack/resources/environment.py", line 340, in wait_until_env_is_up
    sleep=10)
  File "neutron/common/utils.py", line 650, in wait_until_true
    raise WaitTimeout("Timed out after %d seconds" % timeout)
neutron.common.utils.WaitTimeout: Timed out after 180 seconds

and it looks that problem is somewhere in L3 agent which is not even starting in all those tests.

[Slawek Kaplonski (slaweq)]

I was able to reproduce this issue locally on Ubuntu 16.04 with python packages like:

ubuntu@fullstack-master ~ ❯❯❯ dpkg -l | grep python2.7
ii libpython2.7-minimal:amd64 2.7.12-1ubuntu0~16.04.3 amd64 Minimal subset of the Python language (version 2.7)
ii libpython2.7-stdlib:amd64 2.7.12-1ubuntu0~16.04.3 amd64 Interactive high-level object-oriented language (standard library, version 2.7)
ii python2.7 2.7.12-1ubuntu0~16.04.3 amd64 Interactive high-level object-oriented language (version 2.7)
ii python2.7-minimal 2.7.12-1ubuntu0~16.04.3 amd64 Minimal subset of the Python language (version 2.7)

but all is fine when I downgraded python packages to:

ubuntu@fullstack-master ~ ❯❯❯ dpkg -l | grep python2.7
ii libpython2.7-minimal:amd64 2.7.12-1ubuntu0~16.04.2 amd64 Minimal subset of the Python language (version 2.7)
ii libpython2.7-stdlib:amd64 2.7.12-1ubuntu0~16.04.2 amd64 Interactive high-level object-oriented language (standard library, version 2.7)
ii python2.7 2.7.12-1ubuntu0~16.04.2 amd64 Interactive high-level object-oriented language (version 2.7)
ii python2.7-minimal 2.7.12-1ubuntu0~16.04.2 amd64 Minimal subset of the Python language (version 2.7)

[Slawek Kaplonski (slaweq)]

l3_agent.py output in failed fullstack test:

[11:37:35] skaplons@LAPTOP857068 /tmp $ grep "l3_agent.py " TestLegacyL3Agent.test_namespace_exists.txt.gz -ri | grep "output: " | awk -F'output: ' '{print $2}' | awk -F'_watch_process' '{print $1}' | less
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:12> ignored
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:14> ignored
Traceback (most recent call last):
File "neutron/tests/fullstack/cmd/l3_agent.py", line 19, in <module>
from neutron.tests.common.agents import l3_agent
File "/opt/stack/new/neutron/neutron/tests/common/__init__.py", line 16, in <module>
eventlet_utils.monkey_patch()
File "/opt/stack/new/neutron/neutron/common/eventlet_utils.py", line 25, in monkey_patch
p_c_e = importutils.import_module('pyroute2.config.asyncio')
File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/oslo_utils/importutils.py", line 73, in import_module
__import__(import_str)
File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/pyroute2/__init__.py", line 17, in <module>
from pyroute2.netlink.rtnl.req import (IPRouteRequest,
File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/pyroute2/netlink/rtnl/req.py", line 9, in <module>
from pyroute2.netlink.rtnl.ifinfmsg import ifinfmsg
File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/pyroute2/netlink/rtnl/ifinfmsg/__init__.py", line 8, in <module>
from pyroute2 import config
File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/pyroute2/config/__init__.py", line 20, in <module>
arch = platform.architecture()[0]
File "/usr/lib/python2.7/platform.py", line 1087, in architecture
None
output = _syscmd_file(executable, '')
File "/usr/lib/python2.7/platform.py", line 1030, in _syscmd_file
stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
File "/opt/stack/new/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 55, in __init__
subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
File "/usr/lib/python2.7/subprocess.py", line 741, in __init__
self.stdout = os.fdopen(c2pread, 'rb', bufsize)

[Slawek Kaplonski (slaweq)]

It looks that problem is in eventlet version which is used in our requirements (0.20.0).
With eventlet 0.21.0 there is different stack trace reported:

dsvm-fullstack runtests: commands[2] | /opt/stack/neutron/tools/ostestr_compat_shim.sh neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_namespace_exists
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:14> ignored
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:15> ignored
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:8> ignored
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:7> ignored
Exception AttributeError: "'_SocketDuckForFd' object has no attribute '_closed'" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:9> ignored
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/opt/stack/neutron/.tox/dsvm-fullstack/lib/python2.7/site-packages/subunit/run.py", line 149, in <module>
    main()
  File "/opt/stack/neutron/.tox/dsvm-fullstack/lib/python2.7/site-packages/subunit/run.py", line 145, in main
    stdout=stdout, exit=False)
  File "/opt/stack/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/testtools/run.py", line 192, in __init__
    runner.list(self.test, loader=self.testLoader)
  File "/opt/stack/neutron/.tox/dsvm-fullstack/lib/python2.7/site-packages/subunit/run.py", line 77, in list
    result, errors = self._list(test)
  File "/opt/stack/neutron/.tox/dsvm-fullstack/lib/python2.7/site-packages/subunit/run.py", line 95, in _list
    stream = os.fdopen(fileno, 'wb', 0)
  File "/opt/stack/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/green/os.py", line 28, in fdopen
    return greenio.GreenPipe(fd, *args, **kw)
  File "/opt/stack/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/greenio/py2.py", line 41, in __init__
    super(GreenPipe, self).__init__(_SocketDuckForFd(fileno), mode, bufsize)
  File "/opt/stack/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/greenio/py2.py", line 128, in __init__
    notify_opened(fileno)
  File "/opt/stack/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 189, in notify_opened
    hub = get_hub()
  File "/opt/stack/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 116, in get_hub
    use_hub()
  File "/opt/stack/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 70, in use_hub
    mod = get_default_hub()
  File "/opt/stack/neutron/.tox/dsvm-fullstack/local/lib/python2.7/site-packages/eventlet/hubs/__init__.py", line 38, in get_default_hub
    import event...

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/537863

[Slawek Kaplonski (slaweq)]

Similar issue is described also in https://www.bountysource.com/issues/43892421-new-monotonic-broken-on-docker

[Slawek Kaplonski (slaweq)]

So it looks that it is issue in eventlet < 0.22.0 and is fixed/workarounded by commit https://github.com/eventlet/eventlet/commit/b756447bab51046dfc6f1e0e299cc997ab343701

We will apply similar workaround in Neutron for now.

[Slawek Kaplonski (slaweq)]

I reported also bug for Python 2.7 in Ubuntu: https://bugs.launchpad.net/ubuntu/+source/python2.7/+bug/1745475

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/537863
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=04efb3bca33cb4ca49f29e548866ad6f5d0ed359
Submitter: Zuul
Branch: master

commit 04efb3bca33cb4ca49f29e548866ad6f5d0ed359
Author: Sławek Kapłoński <email address hidden>
Date: Thu Jan 25 12:41:38 2018 +0100

    Fix eventlet imports issue

    With Ubuntu python package 2.7.12-1ubuntu0~16.04.3 and
    evetlet < 0.22.0 there is some issue with eventlet monkey_patching.
    This commit adds workaround solution ported from eventlet patch
    https://github.com/eventlet/eventlet/commit/b756447bab51046dfc6f1e0e299cc997ab343701

    Change-Id: Ia381c98116765903bf2652acd175251233df4e54
    Closes-Bug: #1745013

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.0.0rc1 release candidate.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/571683

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/571686

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571683
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=788ae95ff720e08202ac43a122fb35807d522191
Submitter: Zuul
Branch: stable/pike

commit 788ae95ff720e08202ac43a122fb35807d522191
Author: Sławek Kapłoński <email address hidden>
Date: Thu Jan 25 12:41:38 2018 +0100

    Fix eventlet imports issue

    With Ubuntu python package 2.7.12-1ubuntu0~16.04.3 and
    evetlet < 0.22.0 there is some issue with eventlet monkey_patching.
    This commit adds workaround solution ported from eventlet patch
    https://github.com/eventlet/eventlet/commit/b756447bab51046dfc6f1e0e299cc997ab343701

    Change-Id: Ia381c98116765903bf2652acd175251233df4e54
    Closes-Bug: #1745013
    (cherry picked from commit 04efb3bca33cb4ca49f29e548866ad6f5d0ed359)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/571686
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4594ee6913a464a7fb91e23413e104851ee3e09d
Submitter: Zuul
Branch: stable/ocata

commit 4594ee6913a464a7fb91e23413e104851ee3e09d
Author: Sławek Kapłoński <email address hidden>
Date: Thu Jan 25 12:41:38 2018 +0100

    Fix eventlet imports issue

    With Ubuntu python package 2.7.12-1ubuntu0~16.04.3 and
    evetlet < 0.22.0 there is some issue with eventlet monkey_patching.
    This commit adds workaround solution ported from eventlet patch
    https://github.com/eventlet/eventlet/commit/b756447bab51046dfc6f1e0e299cc997ab343701

    Change-Id: Ia381c98116765903bf2652acd175251233df4e54
    Closes-Bug: #1745013
    (cherry picked from commit 04efb3bca33cb4ca49f29e548866ad6f5d0ed359)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 10.0.7 release.