Add enabled_provider_drivers to juju configuration

[Alan Baghumian]

Hello There!

The Octavia OVN provider driver is currently immature and not 100% reliable and occasionally causes Octavia API outage with 500 errors:

[Tue Aug 01 00:05:41.506270 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] octavia_lib.api.drivers.exceptions.DriverAgentNotFound: Unable to open the driver agent socket: /var/run/octavia/status.sock
[Tue Aug 01 00:05:41.506300 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460]
[Tue Aug 01 00:05:41.506326 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] During handling of the above exception, another exception occurred:
[Tue Aug 01 00:05:41.506329 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460]
[Tue Aug 01 00:05:41.506353 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] Traceback (most recent call last):
[Tue Aug 01 00:05:41.506403 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] File "/usr/bin/octavia-wsgi", line 52, in <module>
[Tue Aug 01 00:05:41.506408 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] application = setup_app()
[Tue Aug 01 00:05:41.506416 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 56, in setup_app
[Tue Aug 01 00:05:41.506420 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] _init_drivers()
[Tue Aug 01 00:05:41.506444 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] File "/usr/lib/python3/dist-packages/octavia/api/app.py", line 46, in _init_drivers
[Tue Aug 01 00:05:41.506448 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] driver_factory.get_driver(provider)$ juju run -a octavia 'sudo sed -i s/"enabled_provider_drivers = .*"/"enabled_provider_drivers = amphora:The Octavia Amphora driver"/g /etc/octavia/octavia.conf; sudo systemctl restart octavia-*; sudo systemctl status octavia-*'
[Tue Aug 01 00:05:41.506456 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] File "/usr/lib/python3/dist-packages/octavia/api/drivers/driver_factory.py", line 49, in get_driver
[Tue Aug 01 00:05:41.506460 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] raise exceptions.ProviderNotFound(prov=provider)
[Tue Aug 01 00:05:41.506474 2023] [wsgi:error] [pid 2852319:tid 139662793074432] [remote 127.0.0.1:54460] octavia.common.exceptions.ProviderNotFound: Provider 'ovn' was not found.

It would be great if we have this juju configuration option in order to properly disable this provider driver until it is 100% solid.

Currently we can do this using a juju run, but it will not survive a reboot unless the Octavia config file is made read-only.

Thanks much! Please let me know if there are any questions.

(1) https://storyboard.openstack.org/#!/story/2008710

[Alan Baghumian (alanbach)]

In case anyone needs it:

$ juju run -a octavia 'sudo sed -i s/"enabled_provider_drivers = .*"/"enabled_provider_drivers = amphora:The Octavia Amphora driver"/g /etc/octavia/octavia.conf; sudo systemctl restart octavia-*; sudo systemctl status octavia-*'

[Alex Kavanagh (ajkavanagh)]

Yes, I think this would be a useful addition.

Essentially, add a 'enabled_drivers' config option, or equivalent, to enable the operator to proactively enable drivers. Unfortunately, there will be an overlap with 'enable_amphora', which whilst we can't remove, we should deprecate.

The default for 'enabled_drivers' could initially be "ovn" to be compatible with the current 'enable_amphora' option's default. Then, if enabled_drivers is set to anything other that "ovn", the 'enable_amphora' would be ignored, and just the enabled_drivers used.

[Felipe Reyes (freyes)]

On Thu, 2023-08-03 at 22:05 +0000, Alan Baghumian wrote:
> In case anyone needs it:
>
> $ juju run -a octavia 'sudo sed -i s/"enabled_provider_drivers =
> .*"/"enabled_provider_drivers = amphora:The Octavia Amphora driver"/g
> /etc/octavia/octavia.conf; sudo systemctl restart octavia-*; sudo
> systemctl status octavia-*'
>
how is this different from setting enable-amphora to True?

https://charmhub.io/octavia/configure#enable-amphora

https://opendev.org/openstack/charm-octavia/src/branch/master/src/templates/xena/octavia.conf#L9

[Alan Baghumian (alanbach)]

Hi Felipe,

It's basically the opposite! Hypothetically something along the lines of this:

{% if options.enable_amphora -%}
enabled_provider_drivers = amphora:The Octavia Amphora driver,ovn:Octavia OVN driver
{% else if options.disable_ovn -%}
enabled_provider_drivers = amphora:The Octavia Amphora driver
{% else -%}
enabled_provider_drivers = ovn:Octavia OVN driver
{% endif -%}

[Felipe Reyes (freyes)]

I'm confused now. IIUC you want to have your environment running with the amphora driver instead of ovn, because it's not working as expected, so to get the amphora driver you just need to run "juju config octavia enable-amphora=True", doesn't this give you the results you want?

[Alex Kavanagh (ajkavanagh)]

> I'm confused now. IIUC you want to have your environment running with the amphora driver instead of ovn, because it's not working as expected, so to get the amphora driver you just need to run "juju config octavia enable-amphora=True", doesn't this give you the results you want?

No, sadly not; the ovn driver is enabled *as well*, and if it fails then the octavia service fails as well. This request is to allow selecting which drivers, and only which drivers, the operator wants.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-octavia/+/890834

[Erlon R. Cruz (sombrafam)]

Im as confused as Felipe. If you set enable_amphora=False, disable_ovn=True then you should get
"enabled_provider_drivers = amphora:The Octavia Amphora driver", per Alan's comments on #4. The user experience is terrible but, I think you still can achieve what you want.

[Alex Kavanagh (ajkavanagh)]

> Im as confused as Felipe. If you set enable_amphora=False, disable_ovn=True then you should get
"enabled_provider_drivers = amphora:The Octavia Amphora driver", per Alan's comments on #4. The user experience is terrible but, I think you still can achieve what you want.

@sombrafam where is the 'disable_ovn' option? it doesn't appear in the built charm. Also, there's no enable_amphora option, it's "enable-amphora".

The current master branch template for octavia does this from xena (which is the latest template version):

[api_settings]
{% if options.enable_amphora -%}
enabled_provider_drivers = amphora:The Octavia Amphora driver,ovn:Octavia OVN driver
{% else -%}
enabled_provider_drivers = ovn:Octavia OVN driver
{% endif -%}

i.e. there is no way to switch OFF the ovn driver at the moment. This bug is about switching off the ovn driver completely as if it fails the octavia API exe fails.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-octavia/+/903557

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Erlon R. Cruz <email address hidden>" on branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/charm-octavia/+/903557