Depending on the order that things are started, the ovn-metadata-agent may fail at start up with the following error [0].
This seems to happen because the Chassis is not yet registered (by ovn-controller) in the OVSDB SB database.
I noticed this is more likely to happen when stacking DevStack and services are started one after the other. Perhaps the ovn-metadata-agent should retry fetching it's own chassis from the OVSDB SB database a few times before failing to start as you can see on other logs that the Chassis will appear a few seconds later.
[0]
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron Traceback (most recent call last):
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/bin/neutron-ovn-metadata-agent", line 10, in <module>
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron sys.exit(main())
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/opt/stack/neutron/neutron/cmd/eventlet/agents/ovn_metadata.py", line 24, in main
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron metadata_agent.main()
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/opt/stack/neutron/neutron/agent/ovn/metadata_agent.py", line 39, in main
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron agt.start()
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/opt/stack/neutron/neutron/agent/ovn/metadata/agent.py", line 247, in start
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron self.register_metadata_agent()
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/opt/stack/neutron/neutron/agent/ovn/metadata/agent.py", line 257, in register_metadata_agent
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron self.sb_idl.db_add(table, self.chassis, 'external_ids',
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 42, in execute
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron t.add(self)
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron next(self.gen)
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/api.py", line 110, in transaction
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron del self._nested_txns_map[cur_thread_id]
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/api.py", line 61, in __exit__
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron self.result = self.commit()
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 63, in commit
Nov 25 19:34:45.455716 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron raise result.ex
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 129, in run
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron txn.results.put(txn.do_commit())
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 90, in do_commit
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron command.run_idl(txn)
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 171, in run_idl
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron record = self.api.lookup(self.table, self.record)
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 177, in lookup
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron return self._lookup(table, record)
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/__init__.py", line 224, in _lookup
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron row = idlutils.row_by_value(self, rl.table, rl.column, record)
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron File "/usr/local/lib/python3.8/dist-packages/ovsdbapp/backend/ovs_idl/idlutils.py", line 114, in row_by_value
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron raise RowNotFound(table=table, col=column, match=match)
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron ovsdbapp.backend.ovs_idl.idlutils.RowNotFound: Cannot find Chassis with name=fb530244-3537-4efa-ab6a-7c5c9683c38c
Nov 25 19:34:45.457508 ubuntu-focal-limestone-regionone-0021931462 neutron-ovn-metadata-agent[66875]: ERROR neutron
Depending on the order that things are started, the ovn-metadata-agent may fail at start up with the following error [0].
This seems to happen because the Chassis is not yet registered (by ovn-controller) in the OVSDB SB database.
I noticed this is more likely to happen when stacking DevStack and services are started one after the other. Perhaps the ovn-metadata-agent should retry fetching it's own chassis from the OVSDB SB database a few times before failing to start as you can see on other logs that the Chassis will appear a few seconds later.
[0] focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron Traceback (most recent call last): focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ bin/neutron- ovn-metadata- agent", line 10, in <module> focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron sys.exit(main()) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/opt/stack/ neutron/ neutron/ cmd/eventlet/ agents/ ovn_metadata. py", line 24, in main focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron metadata_ agent.main( ) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/opt/stack/ neutron/ neutron/ agent/ovn/ metadata_ agent.py" , line 39, in main focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron agt.start() focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/opt/stack/ neutron/ neutron/ agent/ovn/ metadata/ agent.py" , line 247, in start focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron self.register_ metadata_ agent() focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/opt/stack/ neutron/ neutron/ agent/ovn/ metadata/ agent.py" , line 257, in register_ metadata_ agent focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron self.sb_ idl.db_ add(table, self.chassis, 'external_ids', focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ backend/ ovs_idl/ command. py", line 42, in execute focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron t.add(self) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/lib/ python3. 8/contextlib. py", line 120, in __exit__ focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron next(self.gen) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ api.py" , line 110, in transaction focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron del self._nested_ txns_map[ cur_thread_ id] focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ api.py" , line 61, in __exit__ focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron self.result = self.commit() focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ backend/ ovs_idl/ transaction. py", line 63, in commit focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron raise result.ex focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ backend/ ovs_idl/ connection. py", line 129, in run focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron txn.results. put(txn. do_commit( )) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ backend/ ovs_idl/ transaction. py", line 90, in do_commit focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron command. run_idl( txn) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ backend/ ovs_idl/ command. py", line 171, in run_idl focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron record = self.api. lookup( self.table, self.record) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ backend/ ovs_idl/ __init_ _.py", line 177, in lookup focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron return self._lookup(table, record) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ backend/ ovs_idl/ __init_ _.py", line 224, in _lookup focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron row = idlutils. row_by_ value(self, rl.table, rl.column, record) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron File "/usr/local/ lib/python3. 8/dist- packages/ ovsdbapp/ backend/ ovs_idl/ idlutils. py", line 114, in row_by_value focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron raise RowNotFound( table=table, col=column, match=match) focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron ovsdbapp. backend. ovs_idl. idlutils. RowNotFound: Cannot find Chassis with name=fb530244- 3537-4efa- ab6a-7c5c9683c3 8c focal-limestone -regionone- 0021931462 neutron- ovn-metadata- agent[66875] : ERROR neutron
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.455716 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-
Nov 25 19:34:45.457508 ubuntu-