[OVN] neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest.test_multiple_ports_portrange_remote often fails

[Maciej Jozefczyk]

We started to see neutron_tempest_plugin.scenario.test_security_groups.NetworkSecGroupTest.test_multiple_ports_portrange_remote failures on:

Example failure:
https://369db73f1617af64c678-28a40d3de53ae200fe2797286e214883.ssl.cf5.rackcdn.com/709110/1/check/neutron-ovn-tempest-ovs-release/0feed71/testr_results.html

Traceback (most recent call last):
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_security_groups.py", line 385, in test_multiple_ports_portrange_remote
    test_ip, port)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_security_groups.py", line 59, in _verify_http_connection
    raise e
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_security_groups.py", line 51, in _verify_http_connection
    ret = utils.call_url_remote(ssh_client, url)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/common/utils.py", line 128, in call_url_remote
    return ssh_client.exec_command(cmd)
  File "/usr/local/lib/python3.6/dist-packages/tenacity/__init__.py", line 311, in wrapped_f
    return self.call(f, *args, **kw)
  File "/usr/local/lib/python3.6/dist-packages/tenacity/__init__.py", line 391, in call
    do = self.iter(retry_state=retry_state)
  File "/usr/local/lib/python3.6/dist-packages/tenacity/__init__.py", line 338, in iter
    return fut.result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 425, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.6/dist-packages/tenacity/__init__.py", line 394, in call
    result = fn(*args, **kwargs)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/common/ssh.py", line 178, in exec_command
    return super(Client, self).exec_command(cmd=cmd, encoding=encoding)
  File "/opt/stack/tempest/tempest/lib/common/ssh.py", line 204, in exec_command
    stderr=err_data, stdout=out_data)
neutron_tempest_plugin.common.utils.SSHExecCommandFailed: Command 'curl http://10.1.0.11:80 --retry 3 --connect-timeout 2' failed, exit status: 28, stderr:
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0Warning: Transient problem: timeout Will retry in 1 seconds. 3 retries left.

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0Warning: Transient problem: timeout Will retry in 2 seconds. 2 retries left.

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0Warning: Transient problem: timeout Will retry in 4 seconds. 1 retries left.

  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
  0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0curl: (28) Connection timed out after 2002 milliseconds

stdout:

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/709762

[Maciej Jozefczyk (maciejjozefczyk)]

I also noticed that it fails even when the 'foo_ok' string is there, but connection was not gracefully closed.

 389 -> pass
(Pdb++) e
Command 'curl http://10.1.0.5:80 --retry 3 --connect-timeout 2' failed, exit status: 56, stderr:
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 7 0 7 0 0 0 0 --:--:-- 0:03:00 --:--:-- 0curl: (56) Recv failure: Connection reset by peer

stdout:
foo_ok

[Maciej Jozefczyk (maciejjozefczyk)]

Root cause has been found, there is missing commit [1] in OVS branch-2.12, from which for now we build OVS and OVN for our gates.

[1] https://github.com/ovn-org/ovn/commit/e659bab31a916d540411c93ca7125011b2e82b5c

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/709762
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4d5817003cc2766839f7629234d7a72697821ae9
Submitter: Zuul
Branch: master

commit 4d5817003cc2766839f7629234d7a72697821ae9
Author: Maciej Józefczyk <email address hidden>
Date: Tue Feb 25 16:58:58 2020 +0100

    Blacklist test_multiple_ports_portrange_remote for OVN

    We recently merged this test and it its very unstable.
    We need to figure out whats wrong. For now lets blacklist
    it to not block the gate.

    Change-Id: I406ad7f839758414903bb66d7480f4ac9975e740
    Related-Bug: #1864620

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

Now we install OVN 20.03.0. This version has the commit mentioned before [1]

The commit reverting https://review.opendev.org/709762 is already merged
https://review.opendev.org/#/c/710516/.

We can now close this bug.

Regards.

[1] https://github.com/ovn-org/ovn/commit/e659bab31a916d540411c93ca7125011b2e82b5c