openstack-dashboard fails hook "dashboard-plugin-relation-change"

[Bas de Bruijne]

In testrun https://solutions.qa.canonical.com/v2/testruns/8efaf515-465e-48b1-a307-b376d235a9a0/, which is yoga jammy with manila, all 3 openstack-dashboard units fail in the "dashboard-plugin-relation-change" hook.

In the logs we see:
=====================
it-openstack-dashboard-2: 03:48:16 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Registered config file: /usr/share/openstack-dashboard/openstack_dashboard/conf/keystonev3_policy.json
unit-openstack-dashboard-2: 03:48:16 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Registered config file: /etc/openstack-dashboard/local_settings.py
unit-openstack-dashboard-2: 03:48:16 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Registered config file: /etc/haproxy/haproxy.cfg
unit-openstack-dashboard-2: 03:48:16 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Registered config file: /etc/apache2/ports.conf
unit-openstack-dashboard-2: 03:48:16 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Registered config file: /etc/apache2/sites-available/000-default.conf
unit-openstack-dashboard-2: 03:48:16 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Registered config file: /etc/apache2/conf-available/openstack-dashboard.conf
unit-openstack-dashboard-2: 03:48:16 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Registered config file: /etc/apache2/sites-available/default-ssl.conf
unit-openstack-dashboard-2: 03:48:16 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Registered config file: /usr/share/openstack-dashboard/openstack_dashboard/enabled/_40_router.py
unit-openstack-dashboard-2: 03:48:17 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Generating template context for identity-service
unit-openstack-dashboard-2: 03:48:17 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Missing required data: service_port service_host
unit-openstack-dashboard-2: 03:48:17 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Missing required data: service_port service_host
unit-openstack-dashboard-2: 03:48:17 INFO unit.openstack-dashboard/2.juju-log dashboard-plugin:475: Missing required data: service_port service_host
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed Traceback (most recent call last):
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/dashboard-plugin-relation-changed", line 555, in <module>
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed main()
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/dashboard-plugin-relation-changed", line 469, in main
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed hooks.execute(sys.argv)
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/core/hookenv.py", line 963, in execute
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed self._hooks[hook_name]()
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/horizon_utils.py", line 787, in wrapped_f
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed f(*args, **kwargs)
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/dashboard-plugin-relation-changed", line 337, in update_plugin_config
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed CONFIGS.write(LOCAL_SETTINGS)
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/templating.py", line 313, in write
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed _out = self.render(config_file).encode('UTF-8')
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/templating.py", line 273, in render
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed ctxt = ostmpl.context()
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/charmhelpers/contrib/openstack/templating.py", line 107, in context
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed _ctxt = context()
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed File "/var/lib/juju/agents/unit-openstack-dashboard-2/charm/hooks/horizon_contexts.py", line 370, in __call__
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed for u, rd in sorted(relations,
unit-openstack-dashboard-2: 03:48:17 WARNING unit.openstack-dashboard/2.dashboard-plugin-relation-changed TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
unit-openstack-dashboard-2: 03:48:17 ERROR juju.worker.uniter.operation hook "dashboard-plugin-relation-changed" (via explicit, bespoke hook script) failed: exit status 1
=====================

Crashdumps and configs can be found here:
https://oil-jenkins.canonical.com/artifacts/8efaf515-465e-48b1-a307-b376d235a9a0/index.html

[Felipe Reyes (freyes)]

there is patch merged in this area on May 30th - https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/869036 - we should check if the revision of the charm deployed contains (or not) this fix.

[Felipe Reyes (freyes)]

Revision 576 effectively has the patch I pointed out.

openstack-dashboard 22.1.0 error 3 openstack-dashboard yoga/stable 576 no hook failed: "dashboard-plugin-relation-changed"

~ $ juju info --series jammy --channel yoga/stable openstack-dashboard
,,,
channels: |
  yoga/stable: 576 2023-05-30 (576) 639kB amd64, arm64, ppc64el, s390x

 ~ $ juju download --series jammy --channel yoga/stable openstack-dashboard

[Bas de Bruijne (basdbruijne)]

This was using 576, which suggests that the patch does not work in this case.

Looking at the patch [0], it now verifies that the `priority` key exists but it does not check that the key is not None, which could still cause this error.

[0]: https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/869036/2/hooks/horizon_contexts.py

[Samuel Allan (samuelallan)]

This has been observed with the manilla-dashboard and octavia-dashboard plugin charms. Sure enough, many charms do set the priority to None - example relation data where we have observed this bug:

```
[('manila-dashboard/0', {'conflicting-packages': [], 'egress-subnets': '10.129.26.62/32', 'ingress-address': '10.129.26.62', 'install-packages': ['python3-manila-ui'], 'local-settings': '', 'priority': None, 'private-address': '10.129.26.62'}), ('octavia-dashboard/0', {'conflicting-packages': ['python3-neutron-lbaas-dashboard'], 'egress-subnets': '10.129.26.62/32', 'ingress-address': '10.129.26.62', 'install-packages': ['python3-octavia-dashboard'], 'local-settings': '', 'priority': None, 'private-address': '10.129.26.62'})]
```

To reproduce, deploy openstack dashboard with these two plugins (manilla and octavia). Note the error status and the traceback relating to comparison between None values - this is due to trying to sort by priority, where the priorities are None.

I think the reason it was working before https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/869036 , is that all the values were strings before. So the comparisons, even with json formatted data, when sorting by priority would have been like `"null" < "1"`.

It looks like the charms mostly incorrectly use the priority (it's supposed to be a string, but I've observed integers and None). Not sure which is the correct source, but everyone I find the definition of this function that sets the priority, it's declared as a string:

- https://github.com/openstack-charmers/charm-interface-dashboard-plugin/blob/master/requires.py#L83
- https://github.com/openstack-charmers/openstack-libs/blob/ab7cc654b3939f0ce09f7458f8a8f2a8801b1627/lib/charms/openstack_libs/v0/dashboard_plugin_requires.py#L204

Possible ideas for fixes:

- convert back to a string after parsing as json (retain old behaviour)
- add more code to check combinations of strings, integers, and None, and try to ensure it's sorted as expected (not sure what exactly is expected though)

As a side note, the patch in question ( https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/869036 ) has some unecessary logic - here, only local-settings and priority are used, yet it decodes other keys too. It would be nice to know why that extra logic is required, also also examples of the values seen with or without json encoding.

This is high priority, as this is blocking deployments now.

[Yoshi Kadokawa (yoshikadokawa)]

We are hitting this issue at customer deployment and due to this the deployment is blocked.
Also no workaround so far.
Subscribing to field-critical

[Alex Kavanagh (ajkavanagh)]

Review for master branch: https://review.opendev.org/c/openstack/charm-openstack-dashboard/+/887594

[Alex Kavanagh (ajkavanagh)]

All backports from 2023.1 to train: https://review.opendev.org/q/topic:bug/2023404+is:open

[Jadon Naas (jadonn)]

A patch for this bug has been merged to master.

[Jadon Naas (jadonn)]

A fix has been merged to stable/2023.1 for this bug.