RBAC Permissions too strict for Chassis_Private table

[Frode Nordahl]

[Impact]
The OpenStack metadata service will not work after upgrade to Hirsute.

[Test Plan]
Execute the gate tests for the neutron-api-plugin-ovn charm, which performs a full cloud deployment and confirms two instances can spawn, get metadata and communicate with each other.

[Regression Potential]
The patch has already been available in the upstream branch-20.12 and has been released in our Focal packages as part of the 20.03.2 point release update for some time.

[Original Bug Description]
After introduction of the Chassis_Private table in OVN 20.09, CMS'es do expect data plane daemons to be able to write to the external_ids column.

However the current RBAC permissions do not allow for this. Running with this patch for ovn-northd fixes the problem:
diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c
index 27df6a379..d332721cd 100644
--- a/northd/ovn-northd.c
+++ b/northd/ovn-northd.c
@@ -12951,7 +12951,7 @@ static const char *rbac_chassis_update[] =
 static const char *rbac_chassis_private_auth[] =
     {"name"};
 static const char *rbac_chassis_private_update[] =
- {"nb_cfg", "nb_cfg_timestamp", "chassis"};
+ {"nb_cfg", "nb_cfg_timestamp", "chassis", "external_ids"};

 static const char *rbac_encap_auth[] =
     {"chassis_name"};

For completeness I will include output from a OpenStack neutron-ovn-metadata-agent daemon when running without the fix:
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command Traceback (most recent call last):
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command t.add(self)
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command next(self.gen)
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 119, in transaction
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command del self._nested_txns_map[cur_thread_id]
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 69, in __exit__
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command self.result = self.commit()
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command raise result.ex
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command txn.results.put(txn.do_commit())
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command raise RuntimeError(msg)
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command RuntimeError: OVSDB Error: {"details":"RBAC rules for client \"ps5-ra4-n2.maas\" role \"ovn-controller\" prohibit modification of table \"Chassis_Private\".","error":"permission error"}
2021-01-25 08:06:51.333 1763580 ERROR ovsdbapp.backend.ovs_idl.command
2021-01-25 08:06:51.334 1763580 CRITICAL neutron [-] Unhandled error: RuntimeError: OVSDB Error: {"details":"RBAC rules for client \"ps5-ra4-n2.maas\" role \"ovn-controller\" prohibit modification of table \"Chassis_Private\".","error":"permission error"}
2021-01-25 08:06:51.334 1763580 ERROR neutron Traceback (most recent call last):
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/bin/neutron-ovn-metadata-agent", line 10, in <module>
2021-01-25 08:06:51.334 1763580 ERROR neutron sys.exit(main())
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/cmd/eventlet/agents/ovn_metadata.py", line 17, in main
2021-01-25 08:06:51.334 1763580 ERROR neutron metadata_agent.main()
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata_agent.py", line 39, in main
2021-01-25 08:06:51.334 1763580 ERROR neutron agt.start()
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/agent.py", line 229, in start
2021-01-25 08:06:51.334 1763580 ERROR neutron self.register_metadata_agent()
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/neutron/agent/ovn/metadata/agent.py", line 239, in register_metadata_agent
2021-01-25 08:06:51.334 1763580 ERROR neutron self.sb_idl.db_add(table, self.chassis, 'external_ids',
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/command.py", line 40, in execute
2021-01-25 08:06:51.334 1763580 ERROR neutron t.add(self)
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__
2021-01-25 08:06:51.334 1763580 ERROR neutron next(self.gen)
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 119, in transaction
2021-01-25 08:06:51.334 1763580 ERROR neutron del self._nested_txns_map[cur_thread_id]
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/api.py", line 69, in __exit__
2021-01-25 08:06:51.334 1763580 ERROR neutron self.result = self.commit()
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 62, in commit
2021-01-25 08:06:51.334 1763580 ERROR neutron raise result.ex
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 122, in run
2021-01-25 08:06:51.334 1763580 ERROR neutron txn.results.put(txn.do_commit())
2021-01-25 08:06:51.334 1763580 ERROR neutron File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/transaction.py", line 118, in do_commit
2021-01-25 08:06:51.334 1763580 ERROR neutron raise RuntimeError(msg)
2021-01-25 08:06:51.334 1763580 ERROR neutron RuntimeError: OVSDB Error: {"details":"RBAC rules for client \"ps5-ra4-n2.maas\" role \"ovn-controller\" prohibit modification of table \"Chassis_Private\".","error":"permission error"}
2021-01-25 08:06:51.334 1763580 ERROR neutron
2021-01-25 08:06:51.375 1763595 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting
2021-01-25 08:06:51.375 1763594 INFO oslo_service.service [-] Parent process has died unexpectedly, exiting
2021-01-25 08:06:51.375 1763595 INFO eventlet.wsgi.server [-] (1763595) wsgi exited, is_accepting=True
2021-01-25 08:06:51.376 1763594 INFO eventlet.wsgi.server [-] (1763594) wsgi exited, is_accepting=True

[Frode Nordahl (fnordahl)]

https://<email address hidden>/

[Brian Murray (brian-murray)]

Hello Frode, or anyone else affected,

Accepted ovn into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ovn/20.12.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Frode, or anyone else affected,

Accepted ovn into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Verified successfully on hirsute-proposed and wallaby-proposed. Please see test results at https://bugs.launchpad.net/cloud-archive/+bug/1914988.

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ovn has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ovn - 20.12.0-0ubuntu3

---------------
ovn (20.12.0-0ubuntu3) hirsute; urgency=medium

  * Add RBAC rules for IGMP_Group table (LP: #1914988):
    - d/p/lp-1914988-Add-IGMP_Group-to-ovn-controller-RBAC.patch
    - d/p/lp-1914988-northd-Add-missing-RBAC-rules-for-FDB-table.patch
    - d/p/lp-1914988-northd-Amend-Chassis-RBAC-rules.patch
    - d/p/lp-1914988-northd-Add-Controller_Event-RBAC-rules.patch
    - d/p/lp-1914988-tests-Amend-release-stale-port-binding-test-for-RBAC.patch
    - d/p/lp-1914988-tests-Use-ovn_start-in-tests-ovn-controller.at.patch
    - d/p/lp-1914988-tests-Make-certificate-generation-extendable.patch
    - d/p/lp-1914988-tests-Test-with-SSL-and-RBAC-for-controller-by-defau.patch
  * d/p/lp-1943266-physical-do-not-forward-traffic-from-localport-to-a-.patch:
    Do not forward traffic from localport to localnet ports (LP: #1943266).
  * d/p/lp-1913024-northd-Add-Chassis_Private-external_ids-column-to-RB.patch
    Update RBAC rules for Chassis_Private table (LP: #1913024).
  * d/p/lp-1917475-northd-Amend-RBAC-rules-for-Port_Binding-table.patch
    Update RBAC rules for Port_Binding table (LP: #1917475).

 -- Frode Nordahl <email address hidden> Fri, 01 Oct 2021 09:42:00 +0200

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ovn has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ovn - 20.12.0-0ubuntu3~cloud0
---------------

 ovn (20.12.0-0ubuntu3~cloud0) focal-wallaby; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ovn (20.12.0-0ubuntu3) hirsute; urgency=medium
 .
   * Add RBAC rules for IGMP_Group table (LP: #1914988):
     - d/p/lp-1914988-Add-IGMP_Group-to-ovn-controller-RBAC.patch
     - d/p/lp-1914988-northd-Add-missing-RBAC-rules-for-FDB-table.patch
     - d/p/lp-1914988-northd-Amend-Chassis-RBAC-rules.patch
     - d/p/lp-1914988-northd-Add-Controller_Event-RBAC-rules.patch
     - d/p/lp-1914988-tests-Amend-release-stale-port-binding-test-for-RBAC.patch
     - d/p/lp-1914988-tests-Use-ovn_start-in-tests-ovn-controller.at.patch
     - d/p/lp-1914988-tests-Make-certificate-generation-extendable.patch
     - d/p/lp-1914988-tests-Test-with-SSL-and-RBAC-for-controller-by-defau.patch
   * d/p/lp-1943266-physical-do-not-forward-traffic-from-localport-to-a-.patch:
     Do not forward traffic from localport to localnet ports (LP: #1943266).
   * d/p/lp-1913024-northd-Add-Chassis_Private-external_ids-column-to-RB.patch
     Update RBAC rules for Chassis_Private table (LP: #1913024).
   * d/p/lp-1917475-northd-Amend-RBAC-rules-for-Port_Binding-table.patch
     Update RBAC rules for Port_Binding table (LP: #1917475).