RBAC Permissions too strict for Port_Binding table

[Liam Young]

[Impact]
The OpenStack Octavia service will not work after upgrade to Hirsute.

[Test Plan]
Execute the gate tests for the octavia charm, which performs a full cloud deployment and confirms successful creation and operation of load balancer.

[Regression Potential]
The patch has already been available in the upstream branch-20.12 and has been released in our Focal packages as part of the 20.03.2 point release update for some time.

[Original Bug Description]
When using Openstack Ussuri with OVN 20.03 and adding a floating IP address to a unbound port the ovn-controller on the hypervisor repeatedly reports:

2021-03-02T10:33:35.517Z|35359|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"juju-eab186-zaza-d26c8c079cc7-11.project.serverstack\" role \"ovn-controller\" prohibit modification of table \"Port_Binding\".","error":"permission error"}
2021-03-02T10:33:35.518Z|35360|main|INFO|OVNSB commit failed, force recompute next time.

The seams to be because the ovn-controller needs to update the virtual_parent attribute of the port binding *2 but that is not included in the list of permissions allowed by the ovn-controller role *1

*1 https://github.com/ovn-org/ovn/blob/aa8ef5588c119fa8615d78288a7db7e3df2d6fbe/northd/ovn-northd.c#L11331-L11332
*2 https://pastebin.ubuntu.com/p/4CfcxgDgdm/

Disabling rbac by changing the role to "" and stopping and starting the southbound db listener results in the port being immediately updated and the floating IP can be accessed.

[Frode Nordahl (fnordahl)]

https://patchwork.ozlabs.org/project/ovn/list/?series=232350

[Frode Nordahl (fnordahl)]

Fixes has been applied upstream for all versions of OVN and we are awaiting upstream to cut point releases to get these and other updates into Ubuntu. We are also working on extending the upstream tests to encompass testing with RBAC by default.

While waiting for that I have picked the relevant fixes into a package provided through a PPA [0].

0: https://launchpad.net/~fnordahl/+archive/ubuntu/lp1917475

[Camille Rodriguez (camille.rodriguez)]

To confirm this is the bug in /var/log/ovn/ovn-controller.log on the hypervisors look for:.

2021-03-02T10:33:35.517Z|35359|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"juju-eab186-zaza-d26c8c079cc7-11.project.serverstack\" role \"ovn-controller\" prohibit modification of table \"Port_Binding\".","error":"permission error"}
2021-03-02T10:33:35.518Z|35360|main|INFO|OVNSB commit failed, force recompute next time.

To disabel rbac, on an ovn-central unit:

# sudo ovn-sbctl find connection
_uuid : a3b68994-4376-4506-81eb-e23d15641305
external_ids : {}
inactivity_probe : 60000
is_connected : false
max_backoff : []
other_config : {}
read_only : false
role : ""
status : {}
target : "pssl:16642"

_uuid : ee53c2b6-ed8b-4b21-9825-a4ecaf2bdc95
external_ids : {}
inactivity_probe : 60000
is_connected : false
max_backoff : []
other_config : {}
read_only : false
role : ovn-controller
status : {}
target : "pssl:6642"

Look for the 6642 listeners uuid. In this case 'ee53c2b6-ed8b-4b21-9825-a4ecaf2bdc95'

Remove the role to disable rbac:

# sudo ovn-sbctl set connection ee53c2b6-ed8b-4b21-9825-a4ecaf2bdc95 role=''

Restart the ovn-controller service on the hypervisors.

To reenable rbac:

# sudo ovn-sbctl set connection e0cef788-df18-4b1b-a238-e8b79ea51c7c role='ovn-controller'

[Frode Nordahl (fnordahl)]

Thank you for adding the extended detail, Camille!

I would like to note that the fix for this is now in -proposed on Focal and is just around the corner to be promoted to -updates. The SRU can be tracked in bug 1924981.

[Giuseppe Petralia (peppepetra)]

I can confirm that on Bionic upgrading to 20.03.2-0ubuntu0.20.04.1~cloud0 fixed this issue

[Dariusz Smigiel (smigiel-dariusz)]

I had exactly the same issue right now on Focal with 20.03.2-0ubuntu0.20.04.1
3 of 6 ovn-controller nodes were reported as "XXX". After restarting all of failing ones, only 2 of 3 reconnected without issues.
The last one ovn-controller was still having problems. The only thing which worked was a workaround from #4

ubuntu@compute-server-6:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal
ubuntu@compute-server-6:~$ sudo apt-cache policy ovn-common
ovn-common:
  Installed: 20.03.2-0ubuntu0.20.04.1
  Candidate: 20.03.2-0ubuntu0.20.04.1
  Version table:
 *** 20.03.2-0ubuntu0.20.04.1 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     20.03.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

[Frode Nordahl (fnordahl)]

@Dariusz, the RBAC rules are in the ovn-northd binary and is applied to the database. Do you have the updated packages installed on the central nodes and are you sure the ovn-northd and possibly the ovn-sb-ovsdb services have restarted after the package upgrade?

[Brian Murray (brian-murray)]

Hello Liam, or anyone else affected,

Accepted ovn into hirsute-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ovn/20.12.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-hirsute to verification-done-hirsute. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-hirsute. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Liam, or anyone else affected,

Accepted ovn into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andre Ruiz (andre-ruiz)]

I seem to be having this problem on a focal / wallaby deployment, although I don't have that exact message (about prohibit update of port_binding), I only have:

root@srv2dell001p:/var/log/ovn# grep -i perm ovn-controller.log │························2021-10-19T14:03:41.342Z|00076|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"srv2dell001p.oam.prd.infra.sicredi.net\" role \"ovn-controller\" prohibit row insertion into table \"Encap\".","│························
error":"permission error"} │························2021-10-19T14:03:41.342Z|00079|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"srv2dell001p.oam.prd.infra.sicredi.net\" role \"ovn-controller\" prohibit row insertion into table \"Chassis\"."│························
,"error":"permission error"} │························2021-10-19T14:03:41.343Z|00081|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"srv2dell001p.oam.prd.infra.sicredi.net\" role \"ovn-controller\" prohibit row insertion into table \"Encap\".","│························
error":"permission error"} │························2021-10-19T14:03:41.344Z|00083|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"srv2dell001p.oam.prd.infra.sicredi.net\" role \"ovn-controller\" prohibit row insertion into table \"Chassis\"."│························
,"error":"permission error"} │························2021-10-19T14:03:41.345Z|00085|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"srv2dell001p.oam.prd.infra.sicredi.net\" role \"ovn-controller\" prohibit row insertion into table \"Chassis\"."│························
,"error":"permission error"}

I'm trying to apply the workaround but the ovn-sbctl is not connecting to the ovndb. Working on that.

Meanwhile, is this considered fixed and released in focal + wallaby?

[Frode Nordahl (fnordahl)]

Andre, we are currently in the bit odd situation where it is fix released for focal but only fix committed for hirsute/focal-wallaby. The good news is that the fix is available in -proposed.

[Andre Ruiz (andre-ruiz)]

Ok, I'll try to update from proposed and test. Thank you!

[Andre Ruiz (andre-ruiz)]

Just upgrading the packages (from focal-wallaby-proposed) did not help. I upgraded on all ovn-chassis (even the octavia ones), all ovn-central, all ovn-chassis-gateway. I also deleted the LB and recreated completely.

On a separate note, when I try to run "ovn-sbctl find connection" the command freezes. Strace shows repeatedly:

poll([{fd=3, events=POLLIN}], 1, 4000) = 0 (Timeout)
getrusage(RUSAGE_THREAD, {ru_utime={tv_sec=0, tv_usec=0}, ru_stime={tv_sec=0, tv_usec=8964}, ...}) = 0
socket(AF_UNIX, SOCK_STREAM, 0) = 5
fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(5, {sa_family=AF_UNIX, sun_path="/var/run/ovn/ovnsb_db.sock"}, 29) = -1 ENOENT (No such file or directory)
close(5)

Any advice is welcome, thank you.

[Frode Nordahl (fnordahl)]

The RBAC rules are installed into the database by ovn-northd on the central units. Depending on which order you upgraded the packages you may need to force the controllers to reconnect.

As for ovn-*ctl hanging, that is a sign you are attempting to talk to a non-leader instance of the database. Take a look at https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-ovn.html#usage for information on how to determine which ovn-central unit is the current leader of the database you want to control.

[Andre Ruiz (andre-ruiz)]

Just a comment on wallaby-proposed packages, I installed those on all ovn-related units and don't see errors about RBAC anymore, and I also didn't notice any other collateral effect.

[Corey Bryant (corey.bryant)]

Testing for this SRU is running here: https://review.opendev.org/c/openstack/charm-octavia/+/815543

[Corey Bryant (corey.bryant)]

Testing has completed successfully for hirsute-proposed and wallaby-proposed. Test results from "Patchset 5 Nov 02 3:05 PM" of the charm-octavia review above.

focal-wallaby-ha-ovn https://openstack-ci-reports.ubuntu.com/artifacts/d85/815543/5/check/focal-wallaby-ha-ovn/d85d874/ : SUCCESS in 1h 49m 16s (non-voting)

focal-wallaby-ha https://openstack-ci-reports.ubuntu.com/artifacts/339/815543/5/check/focal-wallaby-ha/33995ba/ : SUCCESS in 1h 42m 36s

hirsute-wallaby-ha-ovn https://openstack-ci-reports.ubuntu.com/artifacts/97e/815543/5/check/hirsute-wallaby-ha-ovn/97e404a/ : SUCCESS in 2h 05m 08s (non-voting)

hirsute-wallaby-ha https://openstack-ci-reports.ubuntu.com/artifacts/918/815543/5/check/hirsute-wallaby-ha/91892b3/ : SUCCESS in 1h 45m 18s

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for ovn has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ovn - 20.12.0-0ubuntu3

---------------
ovn (20.12.0-0ubuntu3) hirsute; urgency=medium

  * Add RBAC rules for IGMP_Group table (LP: #1914988):
    - d/p/lp-1914988-Add-IGMP_Group-to-ovn-controller-RBAC.patch
    - d/p/lp-1914988-northd-Add-missing-RBAC-rules-for-FDB-table.patch
    - d/p/lp-1914988-northd-Amend-Chassis-RBAC-rules.patch
    - d/p/lp-1914988-northd-Add-Controller_Event-RBAC-rules.patch
    - d/p/lp-1914988-tests-Amend-release-stale-port-binding-test-for-RBAC.patch
    - d/p/lp-1914988-tests-Use-ovn_start-in-tests-ovn-controller.at.patch
    - d/p/lp-1914988-tests-Make-certificate-generation-extendable.patch
    - d/p/lp-1914988-tests-Test-with-SSL-and-RBAC-for-controller-by-defau.patch
  * d/p/lp-1943266-physical-do-not-forward-traffic-from-localport-to-a-.patch:
    Do not forward traffic from localport to localnet ports (LP: #1943266).
  * d/p/lp-1913024-northd-Add-Chassis_Private-external_ids-column-to-RB.patch
    Update RBAC rules for Chassis_Private table (LP: #1913024).
  * d/p/lp-1917475-northd-Amend-RBAC-rules-for-Port_Binding-table.patch
    Update RBAC rules for Port_Binding table (LP: #1917475).

 -- Frode Nordahl <email address hidden> Fri, 01 Oct 2021 09:42:00 +0200

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ovn has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ovn - 20.12.0-0ubuntu3~cloud0
---------------

 ovn (20.12.0-0ubuntu3~cloud0) focal-wallaby; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 ovn (20.12.0-0ubuntu3) hirsute; urgency=medium
 .
   * Add RBAC rules for IGMP_Group table (LP: #1914988):
     - d/p/lp-1914988-Add-IGMP_Group-to-ovn-controller-RBAC.patch
     - d/p/lp-1914988-northd-Add-missing-RBAC-rules-for-FDB-table.patch
     - d/p/lp-1914988-northd-Amend-Chassis-RBAC-rules.patch
     - d/p/lp-1914988-northd-Add-Controller_Event-RBAC-rules.patch
     - d/p/lp-1914988-tests-Amend-release-stale-port-binding-test-for-RBAC.patch
     - d/p/lp-1914988-tests-Use-ovn_start-in-tests-ovn-controller.at.patch
     - d/p/lp-1914988-tests-Make-certificate-generation-extendable.patch
     - d/p/lp-1914988-tests-Test-with-SSL-and-RBAC-for-controller-by-defau.patch
   * d/p/lp-1943266-physical-do-not-forward-traffic-from-localport-to-a-.patch:
     Do not forward traffic from localport to localnet ports (LP: #1943266).
   * d/p/lp-1913024-northd-Add-Chassis_Private-external_ids-column-to-RB.patch
     Update RBAC rules for Chassis_Private table (LP: #1913024).
   * d/p/lp-1917475-northd-Amend-RBAC-rules-for-Port_Binding-table.patch
     Update RBAC rules for Port_Binding table (LP: #1917475).