dashboards-relation-changed fails with grafana 7/stable snap

[DUFOUR Olivier]

When deploying Grafana for LMA, it deploys just fine.
However when an Openstack deployment is being related to the same Grafana, the unit goes into error with the hook dashboards-relation-changed failing.

The environment :
* MaaS 3.2
* Juju 2.9.38
* Grafana running on Focal with 7/stable snap channel 7.5.17

This is clearly related to the release of 7.5.17 last week on 7/stable (related to #1995279). Using for example, the channel 20.04/stable of the snap, suddenly all the hooks seems to complete properly.
However for security reasons on Grafana 7.2.0, this is not an appropriate workaround.

At first glance, it looks like the charm is unable to use correctly the API from Grafana on "http://localhost:3000/api/folders".

#
# versions of Grafana's snap
#

Current version of Grafana :
ubuntu@grafana-1:~$ snap info grafana
name: grafana
summary: feature rich metrics dashboard and graph editor
publisher: Canonical✓
...
snap-id: ehNKbg99zAxHVPCJX5LobcPRwM1xxq4P
tracking: 7/stable
refresh-date: today at 08:23 UTC
channels:
  latest/stable: 6.7.4 2020-06-04 (36) 56MB -
  latest/candidate: 6.7.4 2023-01-12 (36) 56MB -
  latest/beta: 6.7.4 2023-01-12 (36) 56MB -
  latest/edge: 6.7.4 2023-01-12 (36) 56MB -
  rock/stable: –
  rock/candidate: –
  rock/beta: 8.2.7 2022-08-10 (494) 138MB -
  rock/edge: 9.2.1 2022-10-20 (502) 102MB -
  20.04/stable: 7.2.0 2021-03-18 (465) 111MB -
  20.04/candidate: ↑
  20.04/beta: ↑
  20.04/edge: 7.2.0 2021-03-24 (475) 110MB -
  7/stable: 7.5.17 2023-02-14 (531) 61MB -
  7/candidate: 7.5.17 2023-01-14 (531) 61MB -
  7/beta: 7.5.17 2023-01-14 (531) 61MB -
  7/edge: 7.5.17 2023-01-13 (531) 61MB -
  6/stable: 6.7.4 2020-06-04 (36) 56MB -
  6/candidate: 6.7.4 2020-06-04 (36) 56MB -
  6/beta: 6.7.4 2020-06-04 (36) 56MB -
  6/edge: 6.7.4 2020-06-09 (36) 56MB -
  5/stable: 5.0.0 2020-05-19 (7) 33MB -
  5/candidate: ↑
  5/beta: ↑
  5/edge: ↑
installed: 7.5.17 (531) 61MB -

#
# Traceback of the error encountered
#

2023-02-20 10:58:18 ERROR unit.grafana/0.juju-log server.go:316 dashboards:119: Hook error:
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-grafana-0/charm/reactive/grafana.py", line 1431, in import_dashboards
    success, reason = import_dashboard(dash, remote_app, name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/layer/grafana.py", line 219, in import_dashboard
    dashboard["folderId"] = ensure_and_get_dash_folder(
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/layer/grafana.py", line 164, in ensure_and_get_dash_folder
    result = create_folder(folder_name)
  File "/var/lib/juju/agents/unit-grafana-0/charm/lib/charms/layer/grafana.py", line 142, in create_folder
    r.raise_for_status()
  File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 415 Client Error: Unsupported Media Type for url: http://localhost:3000/api/folders

2023-02-20 10:58:18 WARNING unit.grafana/0.dashboards-relation-changed logger.go:60 Traceback (most recent call last):
2023-02-20 10:58:18 WARNING unit.grafana/0.dashboards-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-grafana-0/charm/hooks/dashboards-relation-changed", line 22, in <module>
2023-02-20 10:58:18 WARNING unit.grafana/0.dashboards-relation-changed logger.go:60 main()
2023-02-20 10:58:18 WARNING unit.grafana/0.dashboards-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/charms/reactive/__init__.py", line 74, in main
...
2023-02-20 10:58:18 WARNING unit.grafana/0.dashboards-relation-changed logger.go:60 File "/var/lib/juju/agents/unit-grafana-0/.venv/lib/python3.8/site-packages/requests/models.py", line 941, in raise_for_status
2023-02-20 10:58:18 WARNING unit.grafana/0.dashboards-relation-changed logger.go:60 raise HTTPError(http_error_msg, response=self)
2023-02-20 10:58:18 WARNING unit.grafana/0.dashboards-relation-changed logger.go:60 requests.exceptions.HTTPError: 415 Client Error: Unsupported Media Type for url: http://localhost:3000/api/folders
2023-02-20 10:58:18 ERROR juju.worker.uniter.operation runhook.go:140 hook "dashboards-relation-changed" (via explicit, bespoke hook script) failed: exit status 1

[DUFOUR Olivier (odufourc)]

subscribed ~field-high

It has an impact on Prodstack6 deployment and a deployment for another customer.
But most likely it concerns any new deployment of LMA for Openstack.

[Nobuto Murata (nobuto)]

Looks like the proposed patch has been merged and pushed to the latest/edge channel. What would be the expected timeline and plan to release it to the stable channel?

[Chi Wai CHAN (raychan96)]

It's being promoted to candidate channel. Can you verify it once again using candidate channel with you complete deployment before we release it to stable channel?

[Andy Wu (qch2012)]

confirmed it is working