Fullstack test neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_north_south_traffic is failing

[Slawek Kaplonski]

Fullstack test test neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_north_south_traffic is failing quite often (probably) due to errors in ovs agent.

Examples of failure:

http://logs.openstack.org/11/662111/21/check/neutron-fullstack/c0f8029/testr_results.html.gz
http://logs.openstack.org/29/664629/1/check/neutron-fullstack/e99aa1c/testr_results.html.gz
http://logs.openstack.org/12/640812/2/check/neutron-fullstack/21f02c4/testr_results.html.gz

In both cases in ovs agent's logs there are errors like:
http://logs.openstack.org/11/662111/21/check/neutron-fullstack/c0f8029/controller/logs/dsvm-fullstack-logs/TestLegacyL3Agent.test_north_south_traffic/neutron-openvswitch-agent--2019-06-18--01-52-58-123455_log.txt.gz?level=ERROR

Logstash query:
http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22neutron.tests.fullstack.test_l3_agent.TestLegacyL3Agent.test_north_south_traffic%20%5C%22%20AND%20message%3A%5C%22...%20FAILED%5C%22

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

The error could be caused by a loaded node. In [1], at 2019-06-18T22:01:23.526Z, we can see a inactivity probe error:
2019-06-18T22:01:23.526Z|00061|reconnect|ERR|tcp:127.0.0.1:59893: no response to inactivity probe after 5 seconds, disconnecting

After this error in [2] we can see the OVS agent failing due to a command retry error.

I would suggest to disable the "of_inactivity_probe" parameter during the fullstack tests, or at least in TestLegacyL3Agent. I'll propose a patch for this.

[1] http://logs.openstack.org/12/640812/2/check/neutron-fullstack/21f02c4/controller/logs/openvswitch/ovsdb-server_log.txt.gz
[2] http://logs.openstack.org/12/640812/2/check/neutron-fullstack/21f02c4/controller/logs/dsvm-fullstack-logs/TestLegacyL3Agent.test_north_south_traffic/neutron-openvswitch-agent--2019-06-18--22-00-21-254815_log.txt.gz#_2019-06-18_22_01_26_770

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/667064

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/667064
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d8078469a8693bfa637e607ed848b918e3048ec9
Submitter: Zuul
Branch: master

commit d8078469a8693bfa637e607ed848b918e3048ec9
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 24 10:41:46 2019 +0000

    Disable "of_inactivity_probe" in fullstack tests

    Disable the inactivity probe during the fullstack tests. This probe
    is not tracked nor controller during these tests and usually make
    them more unstable. As seen in the bug report, in loaded CI nodes,
    sometimes the probe raises an error, stopping the current test:

      reconnect|ERR|tcp:127.0.0.1:56401: no response to inactivity probe
        after 5 seconds, disconnecting

    Change-Id: Id27326d5deae78ebfad1012efb507fde9e6c1453
    Closes-Bug: #1833386

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/679464

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/679464
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7b9cdb0c135bbf989fbf49e3eea2d89650fe4626
Submitter: Zuul
Branch: stable/stein

commit 7b9cdb0c135bbf989fbf49e3eea2d89650fe4626
Author: Rodolfo Alonso Hernandez <email address hidden>
Date: Mon Jun 24 10:41:46 2019 +0000

    Disable "of_inactivity_probe" in fullstack tests

    Disable the inactivity probe during the fullstack tests. This probe
    is not tracked nor controller during these tests and usually make
    them more unstable. As seen in the bug report, in loaded CI nodes,
    sometimes the probe raises an error, stopping the current test:

      reconnect|ERR|tcp:127.0.0.1:56401: no response to inactivity probe
        after 5 seconds, disconnecting

    Change-Id: Id27326d5deae78ebfad1012efb507fde9e6c1453
    Closes-Bug: #1833386
    (cherry picked from commit d8078469a8693bfa637e607ed848b918e3048ec9)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 15.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.3 release.