fullstack security groups test fails because ncat process don't starts

[Slawek Kaplonski]

Sometimes fullstack test neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork fails because "ncat" process don't starts properly:

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "neutron/tests/base.py", line 132, in func
        return f(self, *args, **kwargs)
      File "neutron/tests/fullstack/test_securitygroup.py", line 163, in test_securitygroup
        net_helpers.NetcatTester.TCP)
      File "neutron/tests/fullstack/test_securitygroup.py", line 68, in assert_connection
        self.assertTrue(netcat.test_connectivity())
      File "neutron/tests/common/net_helpers.py", line 509, in test_connectivity
        self.client_process.writeline(testing_string)
      File "neutron/tests/common/net_helpers.py", line 459, in client_process
        self.establish_connection()
      File "neutron/tests/common/net_helpers.py", line 489, in establish_connection
        address=self.address)
      File "neutron/tests/common/net_helpers.py", line 537, in _spawn_nc_in_namespace
        proc = RootHelperProcess(cmd, namespace=namespace)
      File "neutron/tests/common/net_helpers.py", line 288, in __init__
        self._wait_for_child_process()
      File "neutron/tests/common/net_helpers.py", line 321, in _wait_for_child_process
        "in %d seconds" % (self.cmd, timeout)))
      File "neutron/common/utils.py", line 649, in wait_until_true
        raise exception
    RuntimeError: Process ['ncat', u'20.0.0.5', '3333', '-w', '20'] hasn't been spawned in 20 seconds

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/541242

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/541242
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=685596f89c02711410bd6b650a12264ca99145fe
Submitter: Zuul
Branch: master

commit 685596f89c02711410bd6b650a12264ca99145fe
Author: Sławek Kapłoński <email address hidden>
Date: Tue Feb 6 12:19:49 2018 +0100

    [Fullstack] Mark security group test as unstable

    As is described in related bug report this tests is failing
    quite often due to some problem with spawning "ncat" process.

    Change-Id: I0651da07fb786cce82262703bfab8ca30ad4125d
    Related-Bug: 1744402

[Slawek Kaplonski (slaweq)]

After unmark this test as unstable it starts failing from time to time again. For example: http://logs.openstack.org/34/555834/3/check/neutron-fullstack/b74983d/logs/testr_results.html.gz

So I need to dig more into it

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/556155

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/556155
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1e49eab25080ff82002604a8f38eafbfa9a4f9b9
Submitter: Zuul
Branch: master

commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402

[Slawek Kaplonski (slaweq)]

Looks that it happens again: http://logs.openstack.org/22/557122/4/check/neutron-fullstack/185813b/logs/testr_results.html.gz
I will investigate there

[Slawek Kaplonski (slaweq)]

No hits in last few weeks: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%5C%22neutron-fullstack%5C%22%20AND%20message%3A%5C%22hasn't%20been%20spawned%20in%2020%20seconds%5C%22
I will close this bug report now

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/queens
Review: https://review.openstack.org/573621

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/pike
Review: https://review.openstack.org/573624

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/573625

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/573625
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=69cab95e0204c1615f04c6bb4995668edab04f10
Submitter: Zuul
Branch: stable/ocata

commit 69cab95e0204c1615f04c6bb4995668edab04f10
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402
    (cherry picked from commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/573624
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e8a9cc1ea9d058871ef108b72b7c58e81f9003d8
Submitter: Zuul
Branch: stable/pike

commit e8a9cc1ea9d058871ef108b72b7c58e81f9003d8
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402
    (cherry picked from commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/573621
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=92a103bdbaaab7bb31dcac6711a8364ca38a2397
Submitter: Zuul
Branch: stable/queens

commit 92a103bdbaaab7bb31dcac6711a8364ca38a2397
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402
    (cherry picked from commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9)

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/830732

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/830732
Committed: https://opendev.org/openstack/neutron/commit/ba4bfa8842db55463059319138151151f1802575
Submitter: "Zuul (22348)"
Branch: master

commit ba4bfa8842db55463059319138151151f1802575
Author: yatinkarel <email address hidden>
Date: Thu Feb 24 10:42:20 2022 +0530

    [Fullstack] Block until dhcp config is done

    Need to ensure ip is configured in vm before doing other
    netcat connectivity checks. It was missed in initial fix[1].

    [1] https://review.opendev.org/c/openstack/neutron/+/556155

    Related-Bug: #1744402
    Change-Id: Icb2c9c056fa9ab5cf42140e824eee4d1427c8f40

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/830374
Committed: https://opendev.org/openstack/neutron/commit/63a8f964269157c07affce0ff616378d21a0d58c
Submitter: "Zuul (22348)"
Branch: master

commit 63a8f964269157c07affce0ff616378d21a0d58c
Author: yatinkarel <email address hidden>
Date: Tue Feb 22 16:34:26 2022 +0530

    Revert "[Fullstack] Mark security groups test as unstable"

    This reverts commit e748e01656e6d2db1510d85e4c25e21ff394c971.
    The known issue is fixed with [1].

    [1] https://review.opendev.org/c/openstack/neutron/+/830732

    Closes-Bug: #1742401
    Related-Bug: #1744402
    Change-Id: I4f503e89e0aa8e63abcea9c951a973168dcb7148

[Lajos Katona (lajos-katona)]

Since https://review.opendev.org/c/openstack/neutron/+/830374 fullstack's unstable decorator is removed from test_security_groups . (that is ~unmaintained/yoga at least since the tag is removed)