neutron

Fullstack tests are failing because of "OSError: [Errno 22] failed to open netns"

Bug #1820865 reported by Slawek Kaplonski on 2019-03-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Critical	Rodolfo Alonso

Bug Description

Example of failure: http://logs.openstack.org/52/586252/36/check/neutron-fullstack/daab6f7/job-output.txt.gz

Test traceback:
Captured traceback:
2019-03-17 15:28:33.814167 | primary | 2019-03-17 15:28:33.813 | ~~~~~~~~~~~~~~~~~~~
2019-03-17 15:28:33.816364 | primary | 2019-03-17 15:28:33.815 | b'Traceback (most recent call last):'
2019-03-17 15:28:33.818734 | primary | 2019-03-17 15:28:33.818 | b' File "/opt/stack/new/neutron/neutron/tests/base.py", line 174, in func'
2019-03-17 15:28:33.820703 | primary | 2019-03-17 15:28:33.820 | b' return f(self, *args, **kwargs)'
2019-03-17 15:28:33.823297 | primary | 2019-03-17 15:28:33.822 | b' File "/opt/stack/new/neutron/neutron/tests/fullstack/test_l3_agent.py", line 166, in test_mtu_update'
2019-03-17 15:28:33.825622 | primary | 2019-03-17 15:28:33.825 | b' common_utils.wait_until_true(lambda: ip.get_devices())'
2019-03-17 15:28:33.827812 | primary | 2019-03-17 15:28:33.827 | b' File "/opt/stack/new/neutron/neutron/common/utils.py", line 684, in wait_until_true'
2019-03-17 15:28:33.829764 | primary | 2019-03-17 15:28:33.829 | b' while not predicate():'
2019-03-17 15:28:33.831982 | primary | 2019-03-17 15:28:33.831 | b' File "/opt/stack/new/neutron/neutron/tests/fullstack/test_l3_agent.py", line 166, in <lambda>'
2019-03-17 15:28:33.834421 | primary | 2019-03-17 15:28:33.833 | b' common_utils.wait_until_true(lambda: ip.get_devices())'
2019-03-17 15:28:40.262004 | primary | 2019-03-17 15:28:40.261 | b' File "/opt/stack/new/neutron/neutron/agent/linux/ip_lib.py", line 162, in get_devices'
2019-03-17 15:28:40.263952 | primary | 2019-03-17 15:28:40.263 | b' devices = privileged.get_device_names(self.namespace)'
2019-03-17 15:28:40.266330 | primary | 2019-03-17 15:28:40.265 | b' File "/opt/stack/new/neutron/neutron/privileged/agent/linux/ip_lib.py", line 567, in get_device_names'
2019-03-17 15:28:40.268162 | primary | 2019-03-17 15:28:40.267 | b' in get_link_devices(namespace, **kwargs)]'
2019-03-17 15:28:40.270638 | primary | 2019-03-17 15:28:40.269 | b' File "/opt/stack/new/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 241, in _wrap'
2019-03-17 15:28:40.272582 | primary | 2019-03-17 15:28:40.272 | b' return self.channel.remote_call(name, args, kwargs)'
2019-03-17 15:28:40.274822 | primary | 2019-03-17 15:28:40.274 | b' File "/opt/stack/new/neutron/.tox/dsvm-fullstack/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 203, in remote_call'
2019-03-17 15:28:40.276631 | primary | 2019-03-17 15:28:40.276 | b' raise exc_type(*result[2])'
2019-03-17 15:28:40.278636 | primary | 2019-03-17 15:28:40.278 | b'OSError: [Errno 22] failed to open netns'

Logstash query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%20to%20open%20netns%5C%22%20AND%20build_name%3A%5C%22neutron-fullstack%5C%22

Tags:

Miguel Lavalle (minsel) on 2019-03-19

Changed in neutron:
assignee:	nobody → Miguel Lavalle (minsel)

Revision history for this message

Miguel Lavalle (minsel) wrote on 2019-03-24:

Modifying logstash query to get the last 7 days:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22failed%20to%20open%20netns%5C%22%20AND%20build_name%3A%5C%22neutron-fullstack%5C%22&from=7d

Revision history for this message

Miguel Lavalle (minsel) wrote on 2019-03-25:

1) Example of the failure: http://logs.openstack.org/32/643732/5/check/neutron-fullstack/75a6569/job-output.txt#_2019-03-24_09_13_21_174900

2) The failure always occurs calling get_device_names in https://github.com/openstack/neutron/blob/master/neutron/privileged/agent/linux/ip_lib.py#L561. This code was introduced in mid December by this patch: https://review.openstack.org/#/c/618273, that implemented get_device_names with pyroute2.

3) We started to become aware of this problem in January: https://bugs.launchpad.net/neutron/+bug/1812364.

Revision history for this message

Miguel Lavalle (minsel) wrote on 2019-03-25:

It seems we are using pyroute1 0.5.4: http://logs.openstack.org/32/643732/5/check/neutron-fullstack/75a6569/tox/dsvm-fullstack-1.log

This is the latest version, which according to pypi was released in March 3: https://pypi.org/project/pyroute2/0.5.4/#history

To remove the possibility that 0.5.4 introduced or reintroduced the problem: https://review.openstack.org/#/c/647256/

OpenStack Infra (hudson-openstack) on 2019-03-25

Changed in neutron:
status:	Confirmed → In Progress

Revision history for this message

Rodolfo Alonso (rodolfo-alonso-hernandez) wrote on 2019-03-26:

I've seen that, sometimes, the test case tries to open the netspace:
devices = ip.get_devices()
before the L3 agent closes the process to create it.

When, in the test, we check if the netspace exists, the filesystem is updated and this function returns true but we can't open yet the file. What we can do is, in the function "network_namespace_exists", add a conditional option to check if the namespace is ready to be opened.

I'll submit a patch with this change.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-26: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/647721

Changed in neutron:
assignee:	Miguel Lavalle (minsel) → Rodolfo Alonso (rodolfo-alonso-hernandez)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-28: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/647721
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=434b29a2cb131d38ed151e6a1d7f1ab756408198
Submitter: Zuul
Branch: master

commit 434b29a2cb131d38ed151e6a1d7f1ab756408198
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Mar 26 11:51:16 2019 +0000

Check the namespace is ready in test_mtu_update tests

Change-Id: Ifdeac05b485a079e2288b413829bce0e39aef6fd
Closes-Bug: #1820865

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-03-29: Change abandoned on neutron (master)

Change abandoned by Miguel Lavalle (<email address hidden>) on branch: master
Review: https://review.openstack.org/647256
Reason: Bug fixed by https://review.openstack.org/#/c/647721/

Bernard Cafarelli (bcafarel) on 2019-06-19

tags:

added: neutron-proactive-backport-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-13: Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/682005

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-13: Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/682006

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-13: Fix proposed to neutron (stable/queens)

#10

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/682008

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-16: Fix merged to neutron (stable/stein)

#11

Reviewed: https://review.opendev.org/682005
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=9f3c7e14ccfcc75ce3eae30a1db564f74a61e3e2
Submitter: Zuul
Branch: stable/stein

commit 9f3c7e14ccfcc75ce3eae30a1db564f74a61e3e2
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Mar 26 11:51:16 2019 +0000

Check the namespace is ready in test_mtu_update tests

    Change-Id: Ifdeac05b485a079e2288b413829bce0e39aef6fd
    Closes-Bug: #1820865
    (cherry picked from commit 434b29a2cb131d38ed151e6a1d7f1ab756408198)

tags:

added: in-stable-stein

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-18: Fix merged to neutron (stable/queens)

#12

Reviewed: https://review.opendev.org/682008
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=261880aea5dbc6ffb51a7756e3ba940ea37ea3e8
Submitter: Zuul
Branch: stable/queens

commit 261880aea5dbc6ffb51a7756e3ba940ea37ea3e8
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Mar 26 11:51:16 2019 +0000

Check the namespace is ready in test_mtu_update tests

    Conflicts:
        neutron/tests/fullstack/base.py
        neutron/privileged/agent/linux/ip_lib.py
        neutron/tests/fullstack/base.py

    Change-Id: Ifdeac05b485a079e2288b413829bce0e39aef6fd
    Closes-Bug: #1820865
    (cherry picked from commit 434b29a2cb131d38ed151e6a1d7f1ab756408198)

tags:

added: in-stable-queens

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-09-18: Fix merged to neutron (stable/rocky)

#13

Reviewed: https://review.opendev.org/682006
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d140750fe4685dbff18c248da7c92fffca8a0789
Submitter: Zuul
Branch: stable/rocky

commit d140750fe4685dbff18c248da7c92fffca8a0789
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Tue Mar 26 11:51:16 2019 +0000

Check the namespace is ready in test_mtu_update tests

Conflicts:
neutron/tests/fullstack/base.py

    Change-Id: Ifdeac05b485a079e2288b413829bce0e39aef6fd
    Closes-Bug: #1820865
    (cherry picked from commit 434b29a2cb131d38ed151e6a1d7f1ab756408198)