Activity log for bug #1913024

Date Who What changed Old value New value Message
2021-01-25 08:22:35 Frode Nordahl bug added bug
2021-01-25 21:24:09 Frode Nordahl ovn (Ubuntu): status New Triaged
2021-01-25 21:24:15 Frode Nordahl ovn (Ubuntu): importance Undecided High
2021-07-15 07:56:25 Frode Nordahl ovn (Ubuntu): status Triaged Fix Committed
2021-09-08 08:06:16 Frode Nordahl nominated for series Ubuntu Focal
2021-09-08 08:06:16 Frode Nordahl bug task added ovn (Ubuntu Focal)
2021-09-08 08:06:16 Frode Nordahl nominated for series Ubuntu Hirsute
2021-09-08 08:06:16 Frode Nordahl bug task added ovn (Ubuntu Hirsute)
2021-09-08 08:06:16 Frode Nordahl nominated for series Ubuntu Impish
2021-09-08 08:06:16 Frode Nordahl bug task added ovn (Ubuntu Impish)
2021-09-08 08:06:25 Frode Nordahl ovn (Ubuntu Impish): status Fix Committed Fix Released
2021-09-08 08:06:30 Frode Nordahl ovn (Ubuntu Hirsute): status New Triaged
2021-09-08 08:06:34 Frode Nordahl ovn (Ubuntu Focal): status New Fix Released
2021-09-08 08:06:46 Frode Nordahl ovn (Ubuntu Hirsute): importance Undecided High
2021-09-23 09:33:27 Frode Nordahl description After introduction of the Chassis_Private table in OVN 20.09, CMS'es do expect data plane daemons to be able to write to the external_ids column. However the current RBAC permissions do not allow for this. Running with this patch for ovn-northd fixes the problem: diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c index 27df6a379..d332721cd 100644 --- a/northd/ovn-northd.c +++ b/northd/ovn-northd.c @@ -12951,7 +12951,7 @@ static const char *rbac_chassis_update[] = static const char *rbac_chassis_private_auth[] = {"name"}; static const char *rbac_chassis_private_update[] = - {"nb_cfg", "nb_cfg_timestamp", "chassis"}; + {"nb_cfg", "nb_cfg_timestamp", "chassis", "external_ids"}; static const char *rbac_encap_auth[] = {"chassis_name"}; For completeness I will include output from a OpenStack neutron-ovn-metadata-agent daemon when running without the fix: 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command t.add(self) 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 119, in transaction 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command del self._nested_txns_map[cur_thread_id] 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 69, in __exit__ 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command raise result.ex 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: {"details":"RBAC rules for client \"ps5-ra4-n2.maas\" role \"ovn-controller\" prohibit modification of table \"Chassis_Private\".","error":"permission error"} 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command 2021-01-25 08:06:51.334 1763580 CRITICAL neutron [-] Unhandled error: RuntimeError: OVSDB Error: {"details":"RBAC rules for client \"ps5-ra4-n2.maas\" role \"ovn-controller\" prohibit modification of table \"Chassis_Private\".","error":"permission error"} 2021-01-25 08:06:51.334 1763580 ERROR neutron Traceback (most recent call last): 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/bin/neutron-ovn-metadata-agent", line 10, in <module> 2021-01-25 08:06:51.334 1763580 ERROR neutron sys.exit(main()) 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/agents/ovn_metadata.py", line 17, in main 2021-01-25 08:06:51.334 1763580 ERROR neutron metadata_agent.main() 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata_agent.py", line 39, in main 2021-01-25 08:06:51.334 1763580 ERROR neutron agt.start() 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/agent.py", line 229, in start 2021-01-25 08:06:51.334 1763580 ERROR neutron self.register_metadata_agent() 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/agent.py", line 239, in register_metadata_agent 2021-01-25 08:06:51.334 1763580 ERROR neutron self.sb_idl.db_add(table, self.chassis, 'external_ids', 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute 2021-01-25 08:06:51.334 1763580 ERROR neutron t.add(self) 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ 2021-01-25 08:06:51.334 1763580 ERROR neutron next(self.gen) 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 119, in transaction 2021-01-25 08:06:51.334 1763580 ERROR neutron del self._nested_txns_map[cur_thread_id] 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 69, in __exit__ 2021-01-25 08:06:51.334 1763580 ERROR neutron self.result = self.commit() 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2021-01-25 08:06:51.334 1763580 ERROR neutron raise result.ex 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run 2021-01-25 08:06:51.334 1763580 ERROR neutron txn.results.put(txn.do_commit()) 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit 2021-01-25 08:06:51.334 1763580 ERROR neutron raise RuntimeError(msg) 2021-01-25 08:06:51.334 1763580 ERROR neutron RuntimeError: OVSDB Error: {"details":"RBAC rules for client \"ps5-ra4-n2.maas\" role \"ovn-controller\" prohibit modification of table \"Chassis_Private\".","error":"permission error"} 2021-01-25 08:06:51.334 1763580 ERROR neutron 2021-01-25 08:06:51.375 1763595 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-01-25 08:06:51.375 1763594 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-01-25 08:06:51.375 1763595 INFO eventlet.wsgi.server [-] (1763595) wsgi exited, is_accepting=True 2021-01-25 08:06:51.376 1763594 INFO eventlet.wsgi.server [-] (1763594) wsgi exited, is_accepting=True [Impact] The OpenStack metadata service will not work after upgrade to Hirsute. [Test Plan] Execute the gate tests for the neutron-api-plugin-ovn charm, which performs a full cloud deployment and confirms two instances can spawn, get metadata and communicate with each other. [Regression Potential] The patch has already been available in the upstream branch-20.12 and has been released in our Focal packages as part of the 20.03.2 point release update for some time. [Original Bug Description] After introduction of the Chassis_Private table in OVN 20.09, CMS'es do expect data plane daemons to be able to write to the external_ids column. However the current RBAC permissions do not allow for this. Running with this patch for ovn-northd fixes the problem: diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c index 27df6a379..d332721cd 100644 --- a/northd/ovn-northd.c +++ b/northd/ovn-northd.c @@ -12951,7 +12951,7 @@ static const char *rbac_chassis_update[] =  static const char *rbac_chassis_private_auth[] =      {"name"};  static const char *rbac_chassis_private_update[] = - {"nb_cfg", "nb_cfg_timestamp", "chassis"}; + {"nb_cfg", "nb_cfg_timestamp", "chassis", "external_ids"};  static const char *rbac_encap_auth[] =      {"chassis_name"}; For completeness I will include output from a OpenStack neutron-ovn-metadata-agent daemon when running without the fix: 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last): 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command t.add(self) 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen) 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 119, in transaction 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command del self._nested_txns_map[cur_thread_id] 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 69, in __exit__ 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit() 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command raise result.ex 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit()) 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg) 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: {"details":"RBAC rules for client \"ps5-ra4-n2.maas\" role \"ovn-controller\" prohibit modification of table \"Chassis_Private\".","error":"permission error"} 2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command 2021-01-25 08:06:51.334 1763580 CRITICAL neutron [-] Unhandled error: RuntimeError: OVSDB Error: {"details":"RBAC rules for client \"ps5-ra4-n2.maas\" role \"ovn-controller\" prohibit modification of table \"Chassis_Private\".","error":"permission error"} 2021-01-25 08:06:51.334 1763580 ERROR neutron Traceback (most recent call last): 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/bin/neutron-ovn-metadata-agent", line 10, in <module> 2021-01-25 08:06:51.334 1763580 ERROR neutron sys.exit(main()) 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/agents/ovn_metadata.py", line 17, in main 2021-01-25 08:06:51.334 1763580 ERROR neutron metadata_agent.main() 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata_agent.py", line 39, in main 2021-01-25 08:06:51.334 1763580 ERROR neutron agt.start() 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/agent.py", line 229, in start 2021-01-25 08:06:51.334 1763580 ERROR neutron self.register_metadata_agent() 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/agent.py", line 239, in register_metadata_agent 2021-01-25 08:06:51.334 1763580 ERROR neutron self.sb_idl.db_add(table, self.chassis, 'external_ids', 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute 2021-01-25 08:06:51.334 1763580 ERROR neutron t.add(self) 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ 2021-01-25 08:06:51.334 1763580 ERROR neutron next(self.gen) 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 119, in transaction 2021-01-25 08:06:51.334 1763580 ERROR neutron del self._nested_txns_map[cur_thread_id] 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 69, in __exit__ 2021-01-25 08:06:51.334 1763580 ERROR neutron self.result = self.commit() 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit 2021-01-25 08:06:51.334 1763580 ERROR neutron raise result.ex 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run 2021-01-25 08:06:51.334 1763580 ERROR neutron txn.results.put(txn.do_commit()) 2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit 2021-01-25 08:06:51.334 1763580 ERROR neutron raise RuntimeError(msg) 2021-01-25 08:06:51.334 1763580 ERROR neutron RuntimeError: OVSDB Error: {"details":"RBAC rules for client \"ps5-ra4-n2.maas\" role \"ovn-controller\" prohibit modification of table \"Chassis_Private\".","error":"permission error"} 2021-01-25 08:06:51.334 1763580 ERROR neutron 2021-01-25 08:06:51.375 1763595 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-01-25 08:06:51.375 1763594 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting 2021-01-25 08:06:51.375 1763595 INFO eventlet.wsgi.server [-] (1763595) wsgi exited, is_accepting=True 2021-01-25 08:06:51.376 1763594 INFO eventlet.wsgi.server [-] (1763594) wsgi exited, is_accepting=True
2021-09-23 10:19:52 Launchpad Janitor merge proposal linked https://code.launchpad.net/~fnordahl/ubuntu/+source/ovn/+git/ovn/+merge/409046
2021-09-23 10:31:10 Frode Nordahl bug task added cloud-archive
2021-09-23 10:31:21 Frode Nordahl nominated for series cloud-archive/wallaby
2021-09-23 10:31:21 Frode Nordahl bug task added cloud-archive/wallaby
2021-09-23 10:31:37 Frode Nordahl cloud-archive: status New Fix Released
2021-09-23 10:32:17 Frode Nordahl cloud-archive: status Fix Released Fix Committed
2021-10-05 17:47:52 Brian Murray ovn (Ubuntu Hirsute): status Triaged Fix Committed
2021-10-05 17:47:54 Brian Murray bug added subscriber Ubuntu Stable Release Updates Team
2021-10-05 17:47:56 Brian Murray bug added subscriber SRU Verification
2021-10-05 17:47:59 Brian Murray tags ps5 ps5 verification-needed verification-needed-hirsute
2021-10-13 17:00:51 Corey Bryant cloud-archive/wallaby: status New Fix Committed
2021-10-13 17:00:54 Corey Bryant tags ps5 verification-needed verification-needed-hirsute ps5 verification-needed verification-needed-hirsute verification-wallaby-needed
2021-10-27 18:12:19 Corey Bryant cloud-archive: status Fix Committed Fix Released
2021-10-27 18:12:36 Corey Bryant tags ps5 verification-needed verification-needed-hirsute verification-wallaby-needed ps5 verification-done verification-done-hirsute verification-wallaby-done
2021-11-03 00:08:11 Chris Halse Rogers removed subscriber Ubuntu Stable Release Updates Team
2021-11-03 00:09:44 Launchpad Janitor ovn (Ubuntu Hirsute): status Fix Committed Fix Released
2021-11-03 12:32:03 Corey Bryant cloud-archive/wallaby: status Fix Committed Fix Released