Fullstack security groups tests failing with "broken pipe" error

Bug #1871908 reported by Slawek Kaplonski
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Slawek Kaplonski

Bug Description

From time to time it happens that fullstack neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork.test_securitygroup test is failing due to BrokenPipeError exception.

Example of such failure:

ft1.2: neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork.test_securitygroup(ovs-openflow)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.6/site-packages/eventlet/greenio/py3.py", line 141, in write
    written = _original_os.write(self._fileno, view[offset:])
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.6/site-packages/eventlet/green/os.py", line 64, in write
    return __original_write__(fd, st)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_securitygroup.py", line 130, in test_securitygroup
    net_helpers.NetcatTester.TCP)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_securitygroup.py", line 72, in assert_connection
    common_utils.wait_until_true(test_connectivity)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 701, in wait_until_true
    while not predicate():
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_securitygroup.py", line 67, in test_connectivity
    return netcat.test_connectivity()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 542, in test_connectivity
    self.client_process.writeline(testing_string)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 320, in writeline
    self.stdin.flush()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.6/site-packages/eventlet/greenio/py3.py", line 144, in write
    raise IOError(*e.args)
BrokenPipeError: [Errno 32] Broken pipe

https://b63fb0eeeda41f664065-da4d3406a0cf7110a347a18ceb92c153.ssl.cf2.rackcdn.com/708985/8/check/neutron-fullstack/7a919ac/testr_results.html

It seems that problem is, that it may happen that nc client or server process isn't really started yet when it tries to read_stdout or writeline in neutron.tests.common.net_helpers.NetcatTester object.
We should handle such exception properly in tests.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.opendev.org/718781

Changed in neutron:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.opendev.org/718781
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c92950b0eea55b8f16ae4ecea9366dba4419545a
Submitter: Zuul
Branch: master

commit c92950b0eea55b8f16ae4ecea9366dba4419545a
Author: Slawek Kaplonski <email address hidden>
Date: Thu Apr 9 17:05:53 2020 +0200

    [Fullstack] Handle properly BrokenPipeError exception

    NetcatTester class should handle BrokenPipeError exception
    and not raise it to fail test immediately if nc process wasn't
    yet started when it tries first time to read/write something to
    it.

    Change-Id: Ica953cc2038b24c4b3985447b393763912aa6abd
    Closes-Bug: #1871908

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/720690

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/720691

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/720692

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/720693

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/train)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/720690
Reason: As this patch is not compatible with py27 and it isn't really needed in stable branches, where fullstack security group tests are marked as unstable, I'm going to abandon this and not spent too much time on this one.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/stein)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/720693
Reason: As this patch is not compatible with py27 and it isn't really needed in stable branches, where fullstack security group tests are marked as unstable, I'm going to abandon this and not spent too much time on this one.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/rocky)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/720691
Reason: As this patch is not compatible with py27 and it isn't really needed in stable branches, where fullstack security group tests are marked as unstable, I'm going to abandon this and not spent too much time on this one.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/queens)

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/720692
Reason: As this patch is not compatible with py27 and it isn't really needed in stable branches, where fullstack security group tests are marked as unstable, I'm going to abandon this and not spent too much time on this one.

tags: added: neutron-proactive-backport-potential
tags: removed: neutron-proactive-backport-potential
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.