Cinder not exporting metrics to Prometheus

[Boris Lukashev]

I've got prometheus enabled along with elastic and grafana, and the data visible to grafana via Prometheus contains no metrics for *cinder_*. Nova, Neutron, etc are all coming through.
Cinder in this case is both CephRBD and LVM iSCSI (multiple BEs).
Kolla current, wallaby openstack revision.

[Mark Goddard (mgoddard)]

Did you enable Cinder after deploying prometheus? If so, you should deploy prometheus again, since the cinder collector would originally have been disabled.

[Boris Lukashev (rageltman)]

Negative, had cinder enabled at the get-go, along with prometheus.

[Mark Goddard (mgoddard)]

Check the command line args. Otherwise, consider raising with the openstack exporter on github.

[Boris Lukashev (rageltman)]

Where should i be looking specifically - i'm a bit new to the monitoring ecosystem in here, bunches of exporters and collectors running everywhere? Thanks

[Mark Goddard (mgoddard)]

prometheus_openstack_exporter runs on the controllers, and collects metrics from OpenStack which are then scraped by prometheus.

[Angelos Kolaitis (aggkolaitis)]

Starting from Wallaby, I noticed that Kolla Ansible does not deploy Cinder V2 API, which is chosen by default by the OpenStack exporter.

I can see the following logs from the prometheus_openstack_exporter container (with default configs):

```
time="2022-01-04T13:55:48Z" level=error msg="enabling exporter for service volume failed: No suitable endpoint could be found in the service catalog." source="main.go:154"
```

As a solution, I have found that you can set volume_api_version=3 in the openstack exporter clouds.yml config file like so:

```
---
clouds:
  default:
    # ...
    identity_api_version: 3
    volume_api_version: 3
    # ...
```

After restart, openstack-exporter then happily includes cinder metrics in the response. I guess I could submit a patch for this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/823397

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/823397
Committed: https://opendev.org/openstack/kolla-ansible/commit/4410ca7802bb8f95e804312fe8d8f5e1e79ad30d
Submitter: "Zuul (22348)"
Branch: master

commit 4410ca7802bb8f95e804312fe8d8f5e1e79ad30d
Author: Angelos Kolaitis <email address hidden>
Date: Tue Jan 4 16:24:40 2022 +0200

    Use Volume V3 API in OpenStack exporter

    Kolla has removed the Volume V2 API by default since OpenStack Wallaby.
    However, openstack-exporter attempts to use the Volume V2 API by
    default, resulting in clean installs failing to fetch Cinder metrics
    in Prometheus.

    This patch updates the clouds.yml configuration file for
    openstack-exporter to use the Volume V3 API instead.

    Closes-Bug: #1938194
    Change-Id: Ifbb601be3ef1a1e853d5a7e832adf556c0ae38b9

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/823832

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/kolla-ansible/+/823833

[Boris Lukashev (rageltman)]

Applied to our Xena environment (deployed as Wallaby initially), ran reconfigure --tags prometheus, and did not see prometheus containers restart nor any changes after manually restarting prometheus_server. Am i applying the change correctly?

[Mark Goddard (mgoddard)]

Hi Boris, you should see the following task run:

Restart prometheus-openstack-exporter container

[Mark Goddard (mgoddard)]

Perhaps you forgot to pip install kolla-ansible after git pull?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/823833
Committed: https://opendev.org/openstack/kolla-ansible/commit/4515dc1508ee2f3f054ab7ca1c8e71aa86a33b93
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 4515dc1508ee2f3f054ab7ca1c8e71aa86a33b93
Author: Angelos Kolaitis <email address hidden>
Date: Tue Jan 4 16:24:40 2022 +0200

    Use Volume V3 API in OpenStack exporter

    Kolla has removed the Volume V2 API by default since OpenStack Wallaby.
    However, openstack-exporter attempts to use the Volume V2 API by
    default, resulting in clean installs failing to fetch Cinder metrics
    in Prometheus.

    This patch updates the clouds.yml configuration file for
    openstack-exporter to use the Volume V3 API instead.

    Closes-Bug: #1938194
    Change-Id: Ifbb601be3ef1a1e853d5a7e832adf556c0ae38b9
    (cherry picked from commit 4410ca7802bb8f95e804312fe8d8f5e1e79ad30d)

[Boris Lukashev (rageltman)]

Hi Mark, thanks for pinging back.
I did not see the exporter restart, and i applied the diff manually to kolla-ansible @ 13 (xena/stable, i think it was 0.1 at that point).
I did manually restart the exporter, but no dice. Rerunning reconfigure --tags prometheus currently after pulling/merging the latest xena/stable.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/kolla-ansible/+/823832
Committed: https://opendev.org/openstack/kolla-ansible/commit/75fd5c89499213c43da20134924894a283ba097d
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 75fd5c89499213c43da20134924894a283ba097d
Author: Angelos Kolaitis <email address hidden>
Date: Tue Jan 4 16:24:40 2022 +0200

    Use Volume V3 API in OpenStack exporter

    Kolla has removed the Volume V2 API by default since OpenStack Wallaby.
    However, openstack-exporter attempts to use the Volume V2 API by
    default, resulting in clean installs failing to fetch Cinder metrics
    in Prometheus.

    This patch updates the clouds.yml configuration file for
    openstack-exporter to use the Volume V3 API instead.

    Closes-Bug: #1938194
    Change-Id: Ifbb601be3ef1a1e853d5a7e832adf556c0ae38b9
    (cherry picked from commit 4410ca7802bb8f95e804312fe8d8f5e1e79ad30d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 14.0.0.0rc1 release candidate.

[Juan Pablo Suazo (jsuazo)]

Angelos Kolaitis mentioned he's seen this issue starting from wallaby. I've also encountered this bug in Victoria, and adding the proposed fix solved my issues. Should this fix be proposed to Victoria too ?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 12.4.0 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/kolla-ansible 13.1.0 release.