Fullstack security groups tests failing with "broken pipe" error

[Slawek Kaplonski]

From time to time it happens that fullstack neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork.test_securitygroup test is failing due to BrokenPipeError exception.

Example of such failure:

ft1.2: neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork.test_securitygroup(ovs-openflow)testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.6/site-packages/eventlet/greenio/py3.py", line 141, in write
    written = _original_os.write(self._fileno, view[offset:])
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.6/site-packages/eventlet/green/os.py", line 64, in write
    return __original_write__(fd, st)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 182, in func
    return f(self, *args, **kwargs)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_securitygroup.py", line 130, in test_securitygroup
    net_helpers.NetcatTester.TCP)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_securitygroup.py", line 72, in assert_connection
    common_utils.wait_until_true(test_connectivity)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/common/utils.py", line 701, in wait_until_true
    while not predicate():
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/test_securitygroup.py", line 67, in test_connectivity
    return netcat.test_connectivity()
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 542, in test_connectivity
    self.client_process.writeline(testing_string)
  File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/common/net_helpers.py", line 320, in writeline
    self.stdin.flush()
  File "/home/zuul/src/opendev.org/openstack/neutron/.tox/dsvm-fullstack-gate/lib/python3.6/site-packages/eventlet/greenio/py3.py", line 144, in write
    raise IOError(*e.args)
BrokenPipeError: [Errno 32] Broken pipe

https://b63fb0eeeda41f664065-da4d3406a0cf7110a347a18ceb92c153.ssl.cf2.rackcdn.com/708985/8/check/neutron-fullstack/7a919ac/testr_results.html

It seems that problem is, that it may happen that nc client or server process isn't really started yet when it tries to read_stdout or writeline in neutron.tests.common.net_helpers.NetcatTester object.
We should handle such exception properly in tests.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/718781

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/718781
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=c92950b0eea55b8f16ae4ecea9366dba4419545a
Submitter: Zuul
Branch: master

commit c92950b0eea55b8f16ae4ecea9366dba4419545a
Author: Slawek Kaplonski <email address hidden>
Date: Thu Apr 9 17:05:53 2020 +0200

    [Fullstack] Handle properly BrokenPipeError exception

    NetcatTester class should handle BrokenPipeError exception
    and not raise it to fail test immediately if nc process wasn't
    yet started when it tries first time to read/write something to
    it.

    Change-Id: Ica953cc2038b24c4b3985447b393763912aa6abd
    Closes-Bug: #1871908

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/720690

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/720691

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/720692

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/720693

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/train
Review: https://review.opendev.org/720690
Reason: As this patch is not compatible with py27 and it isn't really needed in stable branches, where fullstack security group tests are marked as unstable, I'm going to abandon this and not spent too much time on this one.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/stein
Review: https://review.opendev.org/720693
Reason: As this patch is not compatible with py27 and it isn't really needed in stable branches, where fullstack security group tests are marked as unstable, I'm going to abandon this and not spent too much time on this one.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/rocky
Review: https://review.opendev.org/720691
Reason: As this patch is not compatible with py27 and it isn't really needed in stable branches, where fullstack security group tests are marked as unstable, I'm going to abandon this and not spent too much time on this one.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Slawek Kaplonski (<email address hidden>) on branch: stable/queens
Review: https://review.opendev.org/720692
Reason: As this patch is not compatible with py27 and it isn't really needed in stable branches, where fullstack security group tests are marked as unstable, I'm going to abandon this and not spent too much time on this one.