IGMP Snooping does not work with RBAC enabled ovn-controllers

[Pedro Guimarães]

[Impact]
It is currently not possible to use Multicast IGMP snooping with OVN in Ubuntu Focal and Hirsute.

[Test Plan]
1. Execute the gate tests for the neutron-api-plugin-ovn charm, which performs a full cloud deployment and confirms two instances can spawn, get metadata and communicate with each other.

2. Enable IGMP snooping and have instances join a multicast group and validate that packets forward and check for RBAC errors.

[Regression Potential]
This is a very small patch that adds a static RBAC rule that ovn-northd writes into the database. The patch has already been available in the upstream branches since February 2021, we have also landed several similar patches upstream which have previously been made available to Focal through point release updates.

[Original Bug Description]
Hi,

I've tested this on both 20.03 and 20.06.

Looking into ovn-architecture.xml: https://github.com/ovn-org/ovn/blob/master/ovn-architecture.7.xml#L2530
It states that once RBAC is enabled, ovn-controllers will have access to some of the tables and that is hardcoded within OVN.

That means once RBAC is enabled, IGMP_Group table is out of reach for ovn-controllers and will cause the following issue:

2021-02-06T17:17:40.916Z|00028|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client "REDACTED" role "ovn-controller" prohibit row insertion into table "IGMP_Group".","error":"permission error"}

Reported on upstream repo: https://github.com/ovn-org/ovn/issues/77

Proposed patch: https://github.com/phvalguima/ovn/commit/3419d9946c51b413f816ceb82372677e4afdbe9d

[Frode Nordahl (fnordahl)]

A package for bionic-ussuri is available for test here: https://launchpad.net/~fnordahl/+archive/ubuntu/dev2/

[Frode Nordahl (fnordahl)]

The proposed patch was reviewed:
https://<email address hidden>/

And has been applied to upstream master, branch-20.12 and branch-20.09.

As such we will get the fix into Hirsute on next refresh of the snapshot, and we should consider a backport to our 20.06 and 20.03 packages.

[James Troup (elmo)]

This is *NOT* fixed released in Focal; the patch (AFAICS) does not appear in 20.03.2.

[Brett Milford (brettmilford)]

Just confirming this hasn't been back ported to 20.03 (Focal/Ussuri) yet.
Also for reference, its not in Hirsute as yet.

We have a customer deployment running up against this with Bionic-Ussuri UCA.

[Frode Nordahl (fnordahl)]

I somehow convinced myself we got this patch into the upstream 20.03.2 point release, and evidently that is not the case. I want to offer my apologies for any confusion and time wasted as a result of that.

As for Hirsute, there are some patches that are in Focal / Groovy but still pending for the Hirsute OVN package. This is unfortunate and the reason for that is that we upstreamed the patches we carried in our packages to the 20.03, 20.06, 20.09 and 20.12 branches, but only got point releases cut for 20.03 and 20.06, and not 20.12.

I will pick the remaining patches for Hirsute and correct that as part of the SRU process for this bug.

[Brian Murray (brian-murray)]

Hello Pedro, or anyone else affected,

Accepted ovn into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ovn/20.12.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Brian Murray (brian-murray)]

Hello Pedro, or anyone else affected,

Accepted ovn into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ovn/20.03.2-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Pedro, or anyone else affected,

Accepted ovn into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Diko Parvanov (dparv)]

What about fix release for bionic-ussuri?

[Corey Bryant (corey.bryant)]

Hello Pedro, or anyone else affected,

Accepted ovn into ussuri-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ussuri-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ussuri-needed to verification-ussuri-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ussuri-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Frode Nordahl (fnordahl)]

If there is a need for it, I have also provided the Focal and Bionic-Ussuri versions of the packages with this fix in a PPA [0] (pending publication).

0: https://launchpad.net/~fnordahl/+archive/ubuntu/lp1914988

[Corey Bryant (corey.bryant)]

Verified successfully on hirsute-proposed and wallaby-proposed.

[Corey Bryant (corey.bryant)]

Verified successfully on focal-proposed. Bionic-proposed is hitting charm issues that I'm trying to figure out.

[Corey Bryant (corey.bryant)]

Verified successfully on ussuri-proposed. Charm issue is getting fixed here if interested in bionic charm fix: https://review.opendev.org/c/openstack/charm-neutron-api-plugin-ovn/+/815716.

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ovn has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ovn - 20.03.2-0ubuntu0.20.04.2

---------------
ovn (20.03.2-0ubuntu0.20.04.2) focal; urgency=medium

  * Add RBAC rules for IGMP_Group table (LP: #1914988):
    - d/p/lp-1914988-Add-IGMP_Group-to-ovn-controller-RBAC.patch
    - d/p/lp-1914988-tests-Use-ovn_start-in-tests-ovn-controller.at.patch
    - d/p/lp-1914988-tests-Make-certificate-generation-extendable.patch
    - d/p/lp-1914988-tests-Test-with-SSL-and-RBAC-for-controller-by-defau.patch
  * d/p/lp-1943266-physical-do-not-forward-traffic-from-localport-to-a-.patch:
    Do not forward traffic from localport to localnet ports (LP: #1943266).a
  * d/p/lp-1937075-ovn-ctl-Fix-stucked-while-do-cluster-db-init.patch:
    Fix issue where clustered database might not be upgraded (LP: #1937075).

 -- Frode Nordahl <email address hidden> Fri, 01 Oct 2021 09:42:00 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package ovn - 20.12.0-0ubuntu3

---------------
ovn (20.12.0-0ubuntu3) hirsute; urgency=medium

  * Add RBAC rules for IGMP_Group table (LP: #1914988):
    - d/p/lp-1914988-Add-IGMP_Group-to-ovn-controller-RBAC.patch
    - d/p/lp-1914988-northd-Add-missing-RBAC-rules-for-FDB-table.patch
    - d/p/lp-1914988-northd-Amend-Chassis-RBAC-rules.patch
    - d/p/lp-1914988-northd-Add-Controller_Event-RBAC-rules.patch
    - d/p/lp-1914988-tests-Amend-release-stale-port-binding-test-for-RBAC.patch
    - d/p/lp-1914988-tests-Use-ovn_start-in-tests-ovn-controller.at.patch
    - d/p/lp-1914988-tests-Make-certificate-generation-extendable.patch
    - d/p/lp-1914988-tests-Test-with-SSL-and-RBAC-for-controller-by-defau.patch
  * d/p/lp-1943266-physical-do-not-forward-traffic-from-localport-to-a-.patch:
    Do not forward traffic from localport to localnet ports (LP: #1943266).
  * d/p/lp-1913024-northd-Add-Chassis_Private-external_ids-column-to-RB.patch
    Update RBAC rules for Chassis_Private table (LP: #1913024).
  * d/p/lp-1917475-northd-Amend-RBAC-rules-for-Port_Binding-table.patch
    Update RBAC rules for Port_Binding table (LP: #1917475).

 -- Frode Nordahl <email address hidden> Fri, 01 Oct 2021 09:42:00 +0200

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ovn has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ovn - 20.12.0-0ubuntu3~cloud0
---------------

 ovn (20.12.0-0ubuntu3~cloud0) focal-wallaby; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ovn (20.12.0-0ubuntu3) hirsute; urgency=medium
 .
   * Add RBAC rules for IGMP_Group table (LP: #1914988):
     - d/p/lp-1914988-Add-IGMP_Group-to-ovn-controller-RBAC.patch
     - d/p/lp-1914988-northd-Add-missing-RBAC-rules-for-FDB-table.patch
     - d/p/lp-1914988-northd-Amend-Chassis-RBAC-rules.patch
     - d/p/lp-1914988-northd-Add-Controller_Event-RBAC-rules.patch
     - d/p/lp-1914988-tests-Amend-release-stale-port-binding-test-for-RBAC.patch
     - d/p/lp-1914988-tests-Use-ovn_start-in-tests-ovn-controller.at.patch
     - d/p/lp-1914988-tests-Make-certificate-generation-extendable.patch
     - d/p/lp-1914988-tests-Test-with-SSL-and-RBAC-for-controller-by-defau.patch
   * d/p/lp-1943266-physical-do-not-forward-traffic-from-localport-to-a-.patch:
     Do not forward traffic from localport to localnet ports (LP: #1943266).
   * d/p/lp-1913024-northd-Add-Chassis_Private-external_ids-column-to-RB.patch
     Update RBAC rules for Chassis_Private table (LP: #1913024).
   * d/p/lp-1917475-northd-Amend-RBAC-rules-for-Port_Binding-table.patch
     Update RBAC rules for Port_Binding table (LP: #1917475).

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ovn has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ovn - 20.03.2-0ubuntu0.20.04.2~cloud0
---------------

 ovn (20.03.2-0ubuntu0.20.04.2~cloud0) bionic-ussuri; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ovn (20.03.2-0ubuntu0.20.04.2) focal; urgency=medium
 .
   * Add RBAC rules for IGMP_Group table (LP: #1914988):
     - d/p/lp-1914988-Add-IGMP_Group-to-ovn-controller-RBAC.patch
     - d/p/lp-1914988-tests-Use-ovn_start-in-tests-ovn-controller.at.patch
     - d/p/lp-1914988-tests-Make-certificate-generation-extendable.patch
     - d/p/lp-1914988-tests-Test-with-SSL-and-RBAC-for-controller-by-defau.patch
   * d/p/lp-1943266-physical-do-not-forward-traffic-from-localport-to-a-.patch:
     Do not forward traffic from localport to localnet ports (LP: #1943266).a
   * d/p/lp-1937075-ovn-ctl-Fix-stucked-while-do-cluster-db-init.patch:
     Fix issue where clustered database might not be upgraded (LP: #1937075).