/var/lib/grafana is mounted as a docker volume causing new plugins not to show up

[Dawud]

In `ansible/roles/grafana/defaults/main.yml` these are the volumes we mount into the container:

```
grafana_default_volumes:
  - "{{ node_config_directory }}/grafana/:{{ container_config_directory }}/:ro"
  - "/etc/localtime:/etc/localtime:ro"
  - "{{ '/etc/timezone:/etc/timezone:ro' if ansible_facts.os_family == 'Debian' else '' }}"
  - "grafana:/var/lib/grafana/"
  - "kolla_logs:/var/log/kolla/"

The problem with mounting the grafana:/var/lib/grafana volume is that the plug-ins folder exists inside of /var/lib/grafana. This means that after the initial install of grafana, any additional plug-ins installed at build time (Like the recent addition of the opensearch plug-in) are not detected by grafana as the grafana volume is mounted and replaces the contents of /var/lib/grafana (where the plug-ins directory is located). For additional plugins to be installed, they would need to be added to the `grafana` docker volume either manually or by using the grafana-cli after starting the container.

This can be fixed by either 1) removing the need for the grafana volume (as dashboards are loaded as code and stored in the database) or 2) Narrowing down the mountpoint of the grafana volumes to something like `/var/lib/grafana/dashboards`

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/899134

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Dawud <email address hidden>" on branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/899134

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Dawud <email address hidden>" on branch: stable/yoga
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/898736

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/899136

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/899136
Committed: https://opendev.org/openstack/kolla-ansible/commit/8962b4081e25e97f500677b0b55e0800d2acf76f
Submitter: "Zuul (22348)"
Branch: master

commit 8962b4081e25e97f500677b0b55e0800d2acf76f
Author: Dawud <email address hidden>
Date: Wed Oct 18 14:05:10 2023 +0100

    Remove the `grafana` volume

    Fixes not being able to add additional plugins at build time due to the
    `grafana` volume being mounted over the existing `/var/lib/grafana`
    directory. This is fixed by copying the dashboards into the container
    from an existing bind mount instead of using the ``grafana`` volume.
    This however leaves behind the volume which should be removed with
    `docker volume rm grafana` or by setting `grafana_remove_old_volume` to
    `True`.

    Closes-Bug: #2039498
    Change-Id: Ibcffa5d8922c470f655f447558d4a9c73b1ba361