neutron

See error "HashRingIsEmpty" when create instance

Bug #2000495 reported by Gavin Guan
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Expired
Undecided
Unassigned

Bug Description

neutron show below errors when create an instance

2022-12-27 10:31:07.227 1205638 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.230 1205637 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.230 1205630 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-17d58c66-64ec-47d9-8cca-4afb0d6e6e83 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.230 1205632 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-586ce94d-e22c-4b71-b25c-ceb44921d4b0 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.230 1205634 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.230 1205639 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.230 1205633 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-3f48b6b3-1b77-4dce-a78d-cdaa0c9e0b57 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.230 1205636 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-e4d88a33-459d-4933-b59d-86cafbbf6fe8 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205638 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205639 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205636 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-4e648561-04c9-4afa-b117-226461807d14 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205637 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205633 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-71a3cffe-7e68-44f7-b31d-aa0d5f8b09f1 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205632 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-add99db6-d820-4a04-9fe6-b08b57884c85 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205630 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205634 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205635 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-bd75e02f-a59f-4193-91f5-7cf7147219ea - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.231 1205631 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.232 1205631 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-46908903-6ee7-4a7d-82da-3934a51e1190 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.232 1205635 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:07.240 1205641 INFO neutron.db.ovn_revision_numbers_db [req-5157728d-d6d9-40ee-8fc4-ef2531f1a16f - - - - -] Successfully bumped revision number for resource 54e9a52c-b0f2-4a48-98df-3875fbdfbc89 (type: ports) to 3
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.transaction [req-5157728d-d6d9-40ee-8fc4-ef2531f1a16f - - - - -] Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl
    entity = self.api.lookup(self.lookup_table, self.entity)
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 207, in lookup
    return super().lookup(table, record, default=default, timeout=timeout,
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup
    return self._lookup(table, record)
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup
    row = idlutils.row_by_value(self, rl.table, rl.column, record)
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
    raise RowNotFound(table=table, col=column, match=match)
ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab

2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command [req-5157728d-d6d9-40ee-8fc4-ef2531f1a16f - - - - -] Error executing command (PgAclAddCommand): ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command t.add(self)
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen)
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 275, in transaction
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command yield t
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen)
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 110, in transaction
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command del self._nested_txns_map[cur_thread_id]
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 61, in __exit__
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit()
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 65, in commit
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command raise result.ex
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit())
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command command.run_idl(txn)
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command entity = self.api.lookup(self.lookup_table, self.entity)
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 207, in lookup
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command return super().lookup(table, record, default=default, timeout=timeout,
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command return self._lookup(table, record)
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command row = idlutils.row_by_value(self, rl.table, rl.column, record)
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command raise RowNotFound(table=table, col=column, match=match)
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab
2022-12-27 10:31:07.254 1205641 ERROR ovsdbapp.backend.ovs_idl.command
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [req-5157728d-d6d9-40ee-8fc4-ef2531f1a16f - - - - -] Maintenance task: Failed to fix resource 1628ebd0-4f0e-4541-b607-0611168ed4c3 (type: security_group_rules): ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last):
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 392, in check_for_inconsistencies
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 231, in _fix_create_update
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_create'](context, n_obj)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2269, in create_security_group_rule
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._process_security_group_rule(rule)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2262, in _process_security_group_rule
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_acl.update_acls_for_security_group(
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/common/ovn/acl.py", line 305, in update_acls_for_security_group
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn.pg_acl_add(**acl, may_exist=True).execute(check_error=True)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance t.add(self)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 275, in transaction
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance yield t
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 110, in transaction
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance del self._nested_txns_map[cur_thread_id]
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 61, in __exit__
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit()
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 65, in commit
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit())
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance command.run_idl(txn)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance entity = self.api.lookup(self.lookup_table, self.entity)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 207, in lookup
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance return super().lookup(table, record, default=default, timeout=timeout,
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance return self._lookup(table, record)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance row = idlutils.row_by_value(self, rl.table, rl.column, record)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RowNotFound(table=table, col=column, match=match)
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab
2022-12-27 10:31:07.254 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.transaction [req-5157728d-d6d9-40ee-8fc4-ef2531f1a16f - - - - -] Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
    txn.results.put(txn.do_commit())
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
    command.run_idl(txn)
  File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl
    entity = self.api.lookup(self.lookup_table, self.entity)
  File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 207, in lookup
    return super().lookup(table, record, default=default, timeout=timeout,
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup
    return self._lookup(table, record)
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup
    row = idlutils.row_by_value(self, rl.table, rl.column, record)
  File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
    raise RowNotFound(table=table, col=column, match=match)
ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab

2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command [req-5157728d-d6d9-40ee-8fc4-ef2531f1a16f - - - - -] Error executing command (PgAclAddCommand): ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command t.add(self)
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen)
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 275, in transaction
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command yield t
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen)
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 110, in transaction
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command del self._nested_txns_map[cur_thread_id]
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 61, in __exit__
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit()
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 65, in commit
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command raise result.ex
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit())
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command command.run_idl(txn)
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command entity = self.api.lookup(self.lookup_table, self.entity)
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 207, in lookup
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command return super().lookup(table, record, default=default, timeout=timeout,
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command return self._lookup(table, record)
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command row = idlutils.row_by_value(self, rl.table, rl.column, record)
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command raise RowNotFound(table=table, col=column, match=match)
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab
2022-12-27 10:31:07.266 1205641 ERROR ovsdbapp.backend.ovs_idl.command
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [req-5157728d-d6d9-40ee-8fc4-ef2531f1a16f - - - - -] Maintenance task: Failed to fix resource e7813967-9b79-4384-b355-bff5c0de88f1 (type: security_group_rules): ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance Traceback (most recent call last):
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 392, in check_for_inconsistencies
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._fix_create_update(admin_context, row)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/maintenance.py", line 231, in _fix_create_update
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance res_map['ovn_create'](context, n_obj)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2269, in create_security_group_rule
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self._process_security_group_rule(rule)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py", line 2262, in _process_security_group_rule
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn_acl.update_acls_for_security_group(
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/common/ovn/acl.py", line 305, in update_acls_for_security_group
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovn.pg_acl_add(**acl, may_exist=True).execute(check_error=True)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance t.add(self)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 275, in transaction
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance yield t
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance next(self.gen)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 110, in transaction
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance del self._nested_txns_map[cur_thread_id]
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 61, in __exit__
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance self.result = self.commit()
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 65, in commit
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise result.ex
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 131, in run
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance txn.results.put(txn.do_commit())
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 93, in do_commit
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance command.run_idl(txn)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/schema/ovn_northbound/commands.py", line 117, in run_idl
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance entity = self.api.lookup(self.lookup_table, self.entity)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 207, in lookup
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance return super().lookup(table, record, default=default, timeout=timeout,
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 208, in lookup
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance return self._lookup(table, record)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 268, in _lookup
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance row = idlutils.row_by_value(self, rl.table, rl.column, record)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance raise RowNotFound(table=table, col=column, match=match)
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Port_Group with name=pg_13f37b64_2c24_48b7_bf76_a04890898cab
2022-12-27 10:31:07.267 1205641 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance
2022-12-27 10:31:07.268 1205641 INFO neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.maintenance [req-5157728d-d6d9-40ee-8fc4-ef2531f1a16f - - - - -] Maintenance task: Synchronization finished (took 0.07 seconds)
2022-12-27 10:31:09.358 1205633 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-71a3cffe-7e68-44f7-b31d-aa0d5f8b09f1 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:09.359 1205632 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-add99db6-d820-4a04-9fe6-b08b57884c85 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:09.359 1205631 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [req-46908903-6ee7-4a7d-82da-3934a51e1190 - - - - -] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:09.363 1205630 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:09.367 1205635 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'ecc0cc97-5640-40e0-8614-0f17047d4ef3'". This should never happen in a normal situation, please check the status of your cluster
2022-12-27 10:31:09.375 1205631 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovsdb_monitor [-] HashRing is empty, error: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster: neutron.common.ovn.exceptions.HashRingIsEmpty: Hash Ring returned empty when hashing "b'a805e3ae-0e47-4908-b1c5-9761ec25b2f0'". This should never happen in a normal situation, please check the status of your cluster

Tags: ovn
Revision history for this message
Gavin Guan (gavinguanj) wrote :

Openstack Yoga installed on Ubuntu 20.04.4 LTS

root@controller:~# openstack network agent list
+--------------------------------------+------------------------------+------------+-------------------+-------+-------+----------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
 +--------------------------------------+------------------------------+------------+-------------------+-------+-------+----------------------------+
| b7ace7b5-5200-4fa3-8e42-56027d97876d | OVN Controller agent | compute1 | | :-) | UP | ovn-controller |
| 4c073bc7-48b0-5483-8fe5-53515336c63c | OVN Metadata agent | compute1 | | :-) | UP | neutron-ovn-metadata-agent |
| d36f4aee-6889-4bc7-b3ea-4fcefa484d9a | OVN Controller agent | compute2 | | :-) | UP | ovn-controller |
| 140b52bd-dbd1-555a-99ed-bbd4860297a5 | OVN Metadata agent | compute2 | | :-) | UP | neutron-ovn-metadata-agent |
| fb2183a2-628e-49f3-9edf-a44efce5b1ef | OVN Controller Gateway agent | controller | | :-) | UP | ovn-controller |
 +--------------------------------------+------------------------------+------------+-------------------+-------+-------+----------------------------+
root@controller:~#

Revision history for this message
Gavin Guan (gavinguanj) wrote :
Download full text (6.7 KiB)

root@controller:~# openstack server list
+--------------------------------------+------------+--------+----------+--------+----------+
| ID | Name | Status | Networks | Image | Flavor |
 +--------------------------------------+------------+--------+----------+--------+----------+
| 93efa680-b725-4aa3-95e6-b0d934fbef9f | demo-ins01 | ERROR | | cirros | m1.small |
 +--------------------------------------+------------+--------+----------+--------+----------+
root@controller:~#
root@controller:~#
root@controller:~# openstack server show 93efa680-b725-4aa3-95e6-b0d934fbef9f
+-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
 +-----------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| add...

Read more...

Bence Romsics (bence-romsics)
tags: added: ovn
Revision history for this message
Bence Romsics (bence-romsics) wrote :

Hi,

Thanks for the bug report! It seems to me we will need some more information to understand what's going on here:

1) Did this environment work before? If yes, what happened when the behavior changed? If not, how did you install it?
2) I believe neutron-server instances should populate the DB table ovn_hash_ring at startup. Do you have any errors in neutron-server logs between startup and the error in the bug report? If yes, what?
3) What is the content of the ovn_hash_ring table in your neutron DB? Is it empty? What happens if you restart neutron-server? Do you have any errors in the log during restart?
4) A vm boot involves lots of smaller operations, can you identify which operation (API call, port plug) triggers the first error? Do other basic operations work in your environment like 'network create' and 'port create'?

Changed in neutron:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.