test connectivity through 2 routers fails in neutron-ovn-tempest-full-multinode-ovs-master job

[Slawek Kaplonski]

Examples of failure:

https://4ec598fcefc6b0367120-6910015cdc6b96c34eca0ab65a68e7f2.ssl.cf5.rackcdn.com/696926/18/check/neutron-ovn-tempest-full-multinode-ovs-master/c1c51ca/testr_results.html

https://c3c4d9b326375e78bcd8-2bdda90a1128cbc54c09909a8150f07c.ssl.cf2.rackcdn.com/737518/7/check/neutron-ovn-tempest-full-multinode-ovs-master/3976f68/testr_results.html

Traceback:

Traceback (most recent call last):
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/test_connectivity.py", line 124, in test_connectivity_through_2_routers
    servers=servers)
  File "/opt/stack/neutron-tempest-plugin/neutron_tempest_plugin/scenario/base.py", line 375, in check_remote_connectivity
    timeout=timeout, pattern=pattern))
  File "/usr/local/lib/python3.6/dist-packages/unittest2/case.py", line 702, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

[Maciej Jozefczyk (maciejjozefczyk)]

The test

neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers

looks like failing all the time.

Whats new, the same test with combination:

OVN_BRANCH: branch-20.03
OVS_BRANCH: branch-2.13

works.

It seems to be regression caused by commit recently merged in OVN master branch.

[Maciej Jozefczyk (maciejjozefczyk)]

I tried to reproduce this bug locally.

I noticed that while it failed on https://review.opendev.org/#/c/740491/13
it failed with those conditionals:
OVN_BRANCH: branch-20.06
OVS_BRANCH: branch-2.13

I noticed that when it fails, two VMs are spawned separately on different computes (one on controller, second one on compute).

[Maciej Jozefczyk (maciejjozefczyk)]

I think I might have found what is wrong.

There looks like it is a race between updating a LRP.

Based on logs when test fails I can find:

Jul 21 08:12:13.837231 ubuntu-bionic-rax-iad-0018525582 neutron-server[6758]: INFO neutron.db.ovn_revision_numbers_db [None req-1d9e3862-10a2-4530-a0b6-8aa17239a15e tempest-NetworkConnectivityTest-213548775 tempest-NetworkConnectivityTest-
213548775] Successfully bumped revision number for resource 1891206b-e402-41b5-822a-3336e41140af (type: router_ports) to 4
Jul 21 08:12:13.858513 ubuntu-bionic-rax-iad-0018525582 neutron-server[6758]: DEBUG oslo_concurrency.lockutils [None req-da85326a-4d38-4d8e-aff7-47c91e2bb87c None None] Lock "event-dispatch" acquired by "neutron.plugins.ml2.ovo_rpc._Object
ChangeHandler.dispatch_events" :: waited 0.000s {{(pid=7139) inner /usr/local/lib/python3.6/dist-packages/oslo_concurrency/lockutils.py:359}}
Jul 21 08:12:13.879030 ubuntu-bionic-rax-iad-0018525582 neutron-server[6758]: DEBUG neutron_lib.utils.helpers [None req-1d9e3862-10a2-4530-a0b6-8aa17239a15e tempest-NetworkConnectivityTest-213548775 tempest-NetworkConnectivityTest-21354877
5] Time-cost: call 2f533d57-b4de-4738-a434-c20a0cc7e528 function apply_funcs start {{(pid=7133) wrapper /usr/local/lib/python3.6/dist-packages/neutron_lib/utils/helpers.py:218}}
Jul 21 08:12:13.879548 ubuntu-bionic-rax-iad-0018525582 neutron-server[6758]: DEBUG neutron_lib.utils.helpers [None req-1d9e3862-10a2-4530-a0b6-8aa17239a15e tempest-NetworkConnectivityTest-213548775 tempest-NetworkConnectivityTest-21354877
5] Time-cost: call 2f533d57-b4de-4738-a434-c20a0cc7e528 function apply_funcs took 0.001s seconds to run {{(pid=7133) wrapper /usr/local/lib/python3.6/dist-packages/oslo_utils/timeutils.py:388}}
Jul 21 08:12:13.881084 ubuntu-bionic-rax-iad-0018525582 neutron-server[6758]: DEBUG ovsdbapp.backend.ovs_idl.transaction [None req-adf40a64-5267-4612-8644-a48e792017a7 None None] Running txn n=1 command(idx=0): CheckRevisionNumberCommand(n
ame=1891206b-e402-41b5-822a-3336e41140af, resource={'id': '1891206b-e402-41b5-822a-3336e41140af', 'name': '', 'network_id': 'beee037b-496f-4ab3-8af1-a0367ae08ddf', 'tenant_id': '65ec86075de44f5da77bfde9d5a171bd', 'mac_address': 'fa:16:3e:a
9:59:c1', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'b9181e58-5482-4d74-b152-f13928b30b41', 'device_owner': 'network:router_interface', 'fixed_ips': [{'subnet_id': '92c13035-b446-430c-a890-72507605b9b9', 'ip_address': '10.10
.200.5'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['96d53aec-4493-4713-950d-9aaab4944871'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': '', 'binding:vif_type':
 'unbound', 'binding:vif_details': {}, 'port_security_enabled': True, 'qos_policy_id': None, 'qos_network_policy_id': None, 'dns_name': '', 'dns_assignment': [{'ip_address': '10.10.200.5', 'hostname': 'host-10-10-200-5', 'fqdn': 'host-10-1
0-200-5.openstackgate.local.'}], 'dns_domain': '', 'resource_request': None, 'tags': [], 'created_at': '2020-07-21T08:12:10Z', 'updated_at': '2020-07-21T08:12:13Z', 'revision_number': 4, 'project_id': '65ec86075de44f5da77bfde9d5a171bd', 'n
etwork': {'id': 'beee0...

[yatin (yatinkarel)]

test_connectivity_through_2_routers is no longer running in neutron-ovn-tempest-full-multinode-ovs-master job, to get it running i have to set tempest_plugins=neutron-tempest-plugin[1]. The test passed once and failed once. And both time instance was launched on the compute node. The job TIMED_OUT as it enabled many tests.
Also issue is not specific to test_connectivity_through_2_routers case but general. And I think it's due to https://bugs.launchpad.net/neutron/+bug/1904117.

Once the other bug is fixed things should improve. Will try to test it with WIP fixes of other bug to see if that fixes it.

[1] https://review.opendev.org/c/openstack/neutron/+/831220

[yatin (yatinkarel)]

<< Will try to test it with WIP fixes of other bug to see if that fixes it.

Tested with WIP patches in [1], and i see success in multiple runs of all the tests, so will close this bug as duplicate as issue is generic and not limited to a specific test mentioned here, feel free to reopen if not true.

[1] https://review.opendev.org/c/openstack/neutron/+/831220