Funtional tests of start and restart services failing 100% times

[Slawek Kaplonski]

Tests like:

neutron.tests.functional.test_server.TestPluginWorker.test_start
neutron.tests.functional.test_server.TestRPCServer.test_restart_rpc_on_sighup_multiple_workers
neutron.tests.functional.test_server.TestWsgiServer.test_restart_wsgi_on_sighup_multiple_workers

are now failing 100% of times since today.

Example of failures: https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_17/679917/1/check/neutron-functional-python27/caf3c2b/testr_results.html.gz

[Slawek Kaplonski (slaweq)]

It looks like it is failing on oslo.service-1.40.1, on oslo.service-1.40.0 tests are passing. And IMO it is caused by https://github.com/openstack/oslo.service/commit/e7dd2916893157854ca0fa5f4215d536151abbb3
I will now check if this is problem on our side or maybe some bug in oslo.service

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/680001

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/680001
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=bacc7abf83f18825a49af2c14cebbeb312615c1d
Submitter: Zuul
Branch: master

commit bacc7abf83f18825a49af2c14cebbeb312615c1d
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 4 21:50:01 2019 +0200

    Make Neutron gate great again

    This is combined patch to fix couple of issues which
    we recently had in gate.

    1. [Functional tests] Fix SIGHUP handling tests

    Tests in neutron.functional.test_server module are testing how
    PluginWorker, WSGIWorker and RPCWorker are handling SIGHUP signal.

    Recently this was changed in Oslo.service with [1] and our tests
    were failing because they were still expecting that after sending
    SIGHUP to the process, stop() and than start() method will be called.

    But as our services uses "mutate" as restart method, since [1] such
    process don't executes stop() and start() after SIGHUP. It now executes
    only reset() method.
    This patch reflects that change in Neutron functional tests.

    2. Veth pair "IFLA_LINK" populated since kernel 4.15.0-60-generic

    Since kernel_version=4.15.0-60-generic, "iproute2" provides the veth
    pair index, even if the pair interface is in other namespace. In
    previous versions, the parameter 'IFLA_LINK' was not present. We need
    to handle both cases

    [1] https://review.opendev.org/#/c/641907/

    Co-Authored-By: Rodolfo Alonso Hernandez <email address hidden>

    Change-Id: I7a3f20a795c89ab1ab037d046a1101cd5c0287d6
    Closes-Bug: #1842659
    Closes-Bug: #1842482

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/684977

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/684977
Committed: https://git.openstack.org/cgit/openstack/networking-sfc/commit/?id=0644862c8beedd902a7d739be524699edc9b61d3
Submitter: Zuul
Branch: master

commit 0644862c8beedd902a7d739be524699edc9b61d3
Author: Slawek Kaplonski <email address hidden>
Date: Thu Sep 26 11:06:41 2019 +0200

    [Functional tests] Fix SIGHUP handling tests

    Tests in networking_sfc.functional.test_server module are testing how
    service is handling SIGHUP signal.

    Recently this was changed in Oslo.service with [1] and our tests
    were failing because they were still expecting that after sending
    SIGHUP to the process, stop() and than start() method will be called.

    But as our services uses "mutate" as restart method, since [1] such
    process don't executes stop() and start() after SIGHUP. It now executes
    only reset() method.
    Similar change was recently done in Neutron's functional tests in [2].
    This patch reflects that change in networking-sfc functional tests.

    [1] https://review.opendev.org/#/c/641907/
    [2] https://review.opendev.org/#/c/680001/

    Change-Id: I22629c59da983f47ef8b1862afb9a62bdfd78b02
    Closes-Bug: #1842659

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/train
Review: https://review.opendev.org/696476

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/696476
Committed: https://git.openstack.org/cgit/openstack/networking-sfc/commit/?id=cd6a4b53e8ff552af4f01428d2596b0bcdf139e0
Submitter: Zuul
Branch: stable/train

commit cd6a4b53e8ff552af4f01428d2596b0bcdf139e0
Author: Slawek Kaplonski <email address hidden>
Date: Thu Sep 26 11:06:41 2019 +0200

    [Functional tests] Fix SIGHUP handling tests

    Tests in networking_sfc.functional.test_server module are testing how
    service is handling SIGHUP signal.

    Recently this was changed in Oslo.service with [1] and our tests
    were failing because they were still expecting that after sending
    SIGHUP to the process, stop() and than start() method will be called.

    But as our services uses "mutate" as restart method, since [1] such
    process don't executes stop() and start() after SIGHUP. It now executes
    only reset() method.
    Similar change was recently done in Neutron's functional tests in [2].
    This patch reflects that change in networking-sfc functional tests.

    [1] https://review.opendev.org/#/c/641907/
    [2] https://review.opendev.org/#/c/680001/

    Change-Id: I22629c59da983f47ef8b1862afb9a62bdfd78b02
    Closes-Bug: #1842659
    (cherry picked from commit 0644862c8beedd902a7d739be524699edc9b61d3)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/702603

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/702603
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b9f4ce07d6d2b0e2ef4c7899a814862a13e07f45
Submitter: Zuul
Branch: stable/stein

commit b9f4ce07d6d2b0e2ef4c7899a814862a13e07f45
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 4 21:50:01 2019 +0200

    [Functional tests] Fix SIGHUP handling tests

    Tests in neutron.functional.test_server module are testing how
    PluginWorker, WSGIWorker and RPCWorker are handling SIGHUP signal.

    Recently this was changed in Oslo.service with [1] and our tests
    were failing because they were still expecting that after sending
    SIGHUP to the process, stop() and than start() method will be called.

    But as our services uses "mutate" as restart method, since [1] such
    process don't executes stop() and start() after SIGHUP. It now executes
    only reset() method.
    This patch reflects that change in Neutron functional tests.

    NOTE: this patch was merged in master combined with another bug
    resolution [2] due to the existing problems in the CI. The other part
    was already backported in [3].

    [1] https://review.opendev.org/#/c/641907/
    [2] https://review.opendev.org/#/c/680001/
    [3] https://review.opendev.org/#/c/680803/

    Change-Id: I7a3f20a795c89ab1ab037d046a1101cd5c0287d6
    Closes-Bug: #1842659
    (cherry picked from commit bacc7abf83f18825a49af2c14cebbeb312615c1d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.1.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-sfc 10.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/networking-sfc 9.0.1 release.