TypeError: Cannot look up record by empty string in check_for_igmp_snoop_support task

[Krzysztof Klimonda]

openstack ussuri deployment

neutron: 16.1.0
ovn: 20.06.2
ovs: 2.13.1

We are seeing the following traceback for one of maintenance tasks:

--8<--8<--8<--
2020-09-18 08:49:11.572 88 ERROR futurist.periodics [req-036b9eae-53cf-4c55-a946-8aa2dc6697c5 - - - - -] Failed to call periodic 'neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support' (it runs every 600.00 seconds): TypeError: Cannot look up record by empty string
2020-09-18 08:49:11.572 88 ERROR futurist.periodics Traceback (most recent call last):
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/futurist/periodics.py", line 293, in run
2020-09-18 08:49:11.572 88 ERROR futurist.periodics work()
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/futurist/periodics.py", line 67, in __call__
2020-09-18 08:49:11.572 88 ERROR futurist.periodics return self.callback(*self.args, **self.kwargs)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/futurist/periodics.py", line 181, in decorator
2020-09-18 08:49:11.572 88 ERROR futurist.periodics return f(*args, **kwargs)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 551, in check_for_igmp_snoop_support
2020-09-18 08:49:11.572 88 ERROR futurist.periodics ovn_const.MCAST_FLOOD_UNREGISTERED: value})))
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2020-09-18 08:49:11.572 88 ERROR futurist.periodics next(self.gen)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 185, in transaction
2020-09-18 08:49:11.572 88 ERROR futurist.periodics yield t
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
2020-09-18 08:49:11.572 88 ERROR futurist.periodics next(self.gen)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/api.py", line 119, in transaction
2020-09-18 08:49:11.572 88 ERROR futurist.periodics del self._nested_txns_map[cur_thread_id]
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/api.py", line 69, in __exit__
2020-09-18 08:49:11.572 88 ERROR futurist.periodics self.result = self.commit()
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2020-09-18 08:49:11.572 88 ERROR futurist.periodics raise result.ex
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2020-09-18 08:49:11.572 88 ERROR futurist.periodics txn.results.put(txn.do_commit())
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 89, in do_commit
2020-09-18 08:49:11.572 88 ERROR futurist.periodics command.run_idl(txn)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/command.py", line 134, in run_idl
2020-09-18 08:49:11.572 88 ERROR futurist.periodics record = self.api.lookup(self.table, self.record)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 107, in lookup
2020-09-18 08:49:11.572 88 ERROR futurist.periodics return self._lookup(table, record)
2020-09-18 08:49:11.572 88 ERROR futurist.periodics File "/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 115, in _lookup
2020-09-18 08:49:11.572 88 ERROR futurist.periodics raise TypeError("Cannot look up record by empty string")
2020-09-18 08:49:11.572 88 ERROR futurist.periodics TypeError: Cannot look up record by empty string
2020-09-18 08:49:11.572 88 ERROR futurist.periodics
2020-09-18 08:49:12.015 88 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [req-036b9eae-53cf-4c55-a946-8aa2dc6697c5 - - - - -] Maintenance task: Synchronization finished (took 0.40 seconds)
--8<--8<--8<--

[Hongbin Lu (hongbin.lu)]

@Krzysztof Klimonda,

It is not clear to me how to reproduce this issue. Are you able to provide reproducing steps? If not, would you provide more details about your "maintenance tasks". What is that? How it configured neutron? What workload it runs?

[Hongbin Lu (hongbin.lu)]

I am going to set the status to "incompleted". Please feel free to reset the status to "new" once more information is provided.

[Krzysztof Klimonda (kklimonda)]

The maintenance task that is failing is mentioned in the traceback: neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance.DBInconsistenciesPeriodics.check_for_igmp_snoop_support

Unfortunately I don't know what is causing that issue - the root cause for the error itself seems to be an entry in Logical_Switch table without any configuration:

```
_uuid : 6ba3f2a7-fb25-481e-a261-a3501c4c1054
acls : []
dns_records : []
external_ids : {}
forwarding_groups : []
load_balancer : []
name : ""
other_config : {}
ports : []
qos_rules : []
```

This entry could have been created during my ovn stress tests, deleting it from Northbound DB fixes that maintenance task, however adding an ERROR-level message could be useful here, or some other task that clears up database from dangling entries?

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello Krzysztof:

This LS should have been created manually; any LS created by Neutron has "name-<net_id>" as name always assigned. This LS should not be present in the DB and should be deleted manually.

Regards.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Patch: https://review.opendev.org/#/c/758391/

[Lucas Alvares Gomes (lucasagomes)]

Fixed by https://review.opendev.org/c/openstack/neutron/+/758391

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.