fullstack security groups test fails because ncat process don't starts

Bug #1744402 reported by Slawek Kaplonski on 2018-01-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
High
Slawek Kaplonski

Bug Description

Sometimes fullstack test neutron.tests.fullstack.test_securitygroup.TestSecurityGroupsSameNetwork fails because "ncat" process don't starts properly:

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "neutron/tests/base.py", line 132, in func
        return f(self, *args, **kwargs)
      File "neutron/tests/fullstack/test_securitygroup.py", line 163, in test_securitygroup
        net_helpers.NetcatTester.TCP)
      File "neutron/tests/fullstack/test_securitygroup.py", line 68, in assert_connection
        self.assertTrue(netcat.test_connectivity())
      File "neutron/tests/common/net_helpers.py", line 509, in test_connectivity
        self.client_process.writeline(testing_string)
      File "neutron/tests/common/net_helpers.py", line 459, in client_process
        self.establish_connection()
      File "neutron/tests/common/net_helpers.py", line 489, in establish_connection
        address=self.address)
      File "neutron/tests/common/net_helpers.py", line 537, in _spawn_nc_in_namespace
        proc = RootHelperProcess(cmd, namespace=namespace)
      File "neutron/tests/common/net_helpers.py", line 288, in __init__
        self._wait_for_child_process()
      File "neutron/tests/common/net_helpers.py", line 321, in _wait_for_child_process
        "in %d seconds" % (self.cmd, timeout)))
      File "neutron/common/utils.py", line 649, in wait_until_true
        raise exception
    RuntimeError: Process ['ncat', u'20.0.0.5', '3333', '-w', '20'] hasn't been spawned in 20 seconds

Changed in neutron:
assignee: nobody → Slawek Kaplonski (slaweq)

Reviewed: https://review.openstack.org/541242
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=685596f89c02711410bd6b650a12264ca99145fe
Submitter: Zuul
Branch: master

commit 685596f89c02711410bd6b650a12264ca99145fe
Author: Sławek Kapłoński <email address hidden>
Date: Tue Feb 6 12:19:49 2018 +0100

    [Fullstack] Mark security group test as unstable

    As is described in related bug report this tests is failing
    quite often due to some problem with spawning "ncat" process.

    Change-Id: I0651da07fb786cce82262703bfab8ca30ad4125d
    Related-Bug: 1744402

Slawek Kaplonski (slaweq) wrote :

After unmark this test as unstable it starts failing from time to time again. For example: http://logs.openstack.org/34/555834/3/check/neutron-fullstack/b74983d/logs/testr_results.html.gz

So I need to dig more into it

Reviewed: https://review.openstack.org/556155
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1e49eab25080ff82002604a8f38eafbfa9a4f9b9
Submitter: Zuul
Branch: master

commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402

tags: added: neutron-proactive-backport-potential
tags: added: neutron-easy-proactive-backport-potential

Reviewed: https://review.openstack.org/573625
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=69cab95e0204c1615f04c6bb4995668edab04f10
Submitter: Zuul
Branch: stable/ocata

commit 69cab95e0204c1615f04c6bb4995668edab04f10
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402
    (cherry picked from commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9)

tags: added: in-stable-ocata
tags: added: in-stable-pike

Reviewed: https://review.openstack.org/573624
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=e8a9cc1ea9d058871ef108b72b7c58e81f9003d8
Submitter: Zuul
Branch: stable/pike

commit e8a9cc1ea9d058871ef108b72b7c58e81f9003d8
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402
    (cherry picked from commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9)

tags: added: in-stable-queens

Reviewed: https://review.openstack.org/573621
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=92a103bdbaaab7bb31dcac6711a8364ca38a2397
Submitter: Zuul
Branch: stable/queens

commit 92a103bdbaaab7bb31dcac6711a8364ca38a2397
Author: Sławek Kapłoński <email address hidden>
Date: Sun Mar 25 10:42:45 2018 +0200

    [Fullstack] Add block test until IPs will be really configured

    In securitygroups fullstack test dhcp agent is used to configure
    IP addresses inside FakeMachines.
    During preparation to test, when such fake vms are created,
    they are only blocked until boot - so until port becomes ACTIVE
    in Neutron's database.
    This patch adds also blocking vms until IP address is really
    configured on each of such VMs.
    If IP address will be not configured then we should at least have
    better error message then error that nc process cannot be started.

    Change-Id: I709b1489132edba932a4c73bea60f810c26011fa
    Related-Bug: #1744402
    (cherry picked from commit 1e49eab25080ff82002604a8f38eafbfa9a4f9b9)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers