Scenario test test_multiple_ports_secgroup_inheritance fails in ovn scenario job

[Slawek Kaplonski]

Failure:

Traceback (most recent call last):
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/test_security_groups.py", line 341, in test_multiple_ports_secgroup_inheritance
    self.ping_ip_address(fip['floating_ip_address'])
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/neutron_tempest_plugin/scenario/base.py", line 449, in ping_ip_address
    self.assertTrue(result)
  File "/opt/stack/tempest/.tox/tempest/lib/python3.8/site-packages/unittest2/case.py", line 702, in assertTrue
    raise self.failureException(msg)
AssertionError: False is not true

Logs: https://bfc2304b36c89dd5efde-d71f4126f88f4263fd488933444cea49.ssl.cf1.rackcdn.com/740569/2/check/neutron-tempest-plugin-scenario-ovn/026535a/testr_results.html

[Jakub Libosvar (libosvar)]

I ran the test over 50 times on my local setup and it's always passed. Currently, there are no OVN logs collected on the gate, I sent this patch: https://review.opendev.org/c/openstack/neutron/+/771658 and I'll try to reproduce the failure on the gate once the patch merges.

[Jakub Libosvar (libosvar)]

All I could find in the Neutron logs is this error related to the router where FIPs are:

Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command [None req-87984c86-e2e4-4284-b0ca-def6533d63f4 tempest-NetworkSecGroupTest-582045150 tempest-NetworkSecGroupTest-582045150] Error executing command: ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-87af08cb-8b37-4c13-926d-4ce74acc6d3a
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 39, in execute
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command self.run_idl(None)
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 328, in run_idl
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command self.result = self.api.lookup(self.table, self.record)
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 177, in lookup
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command return self._lookup(table, record)
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 224, in _lookup
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command row = idlutils.row_by_value(self, rl.table, rl.column, record)
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command raise RowNotFound(table=table, col=column, match=match)
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Logical_Router with name=neutron-87af08cb-8b37-4c13-926d-4ce74acc6d3a
Jan 08 19:05:10.918498 ubuntu-focal-ovh-gra1-0022453056 neutron-server[68548]: ERROR ovsdbapp.backend.ovs_idl.command

Unfortunately it's hard to tell what command fails there as there is no related log line from the same worker process. I'll try to dig more

[Slawek Kaplonski (slaweq)]

I didn't saw it recently in the gate. Lets close it for now.