@Billy:

We've often hit situations where we get alerts regarding cinder services not being online.  I'm not sure of all the causes - an obvious cause would be if a cinder unit were redeployed to a new machine or LXD, but it may happen in other cases as well - perhaps in case of upgrades or service restarts, but I am somewhat speculating here.

Here is one example from one of our internal clouds:


$ cinder service-list
WARNING:cinderclient.shell:API version 3.66 requested, 
WARNING:cinderclient.shell:downgrading to 3.60 based on server support.
+------------------+--------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+
| Binary           | Host                     | Zone | Status  | State | Updated_at                 | Cluster | Disabled Reason | Backend State |
+------------------+--------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+
| cinder-scheduler | cinder                   | nova | enabled | up    | 2024-02-12T12:33:12.000000 | -       | -               |               |
| cinder-volume    | cinder@cinder-ceph-flash | nova | enabled | down  | 2024-02-12T11:07:54.000000 | -       | -               | up            |
| cinder-volume    | cinder@cinder-ceph-flash | nova | enabled | down  | 2024-02-12T11:08:53.000000 | -       | -               | up            |
| cinder-volume    | cinder@cinder-ceph-flash | nova | enabled | up    | 2024-02-12T12:33:11.000000 | -       | -               | up            |
| cinder-volume    | cinder@cinder-ceph-hdd   | nova | enabled | down  | 2024-02-12T11:07:56.000000 | -       | -               | up            |
| cinder-volume    | cinder@cinder-ceph-hdd   | nova | enabled | down  | 2024-02-12T11:09:01.000000 | -       | -               | up            |
| cinder-volume    | cinder@cinder-ceph-hdd   | nova | enabled | up    | 2024-02-12T12:33:09.000000 | -       | -               | up            |
+------------------+--------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+


This action was run to clean up the old records: juju run-action cinder/leader remove-services --wait 


And the resulting cinder service status was:


$ cinder service-list
WARNING:cinderclient.shell:API version 3.66 requested, 
WARNING:cinderclient.shell:downgrading to 3.60 based on server support.
+------------------+--------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+
| Binary           | Host                     | Zone | Status  | State | Updated_at                 | Cluster | Disabled Reason | Backend State |
+------------------+--------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+
| cinder-scheduler | cinder                   | nova | enabled | up    | 2024-02-12T12:35:42.000000 | -       | -               |               |
| cinder-volume    | cinder@cinder-ceph-flash | nova | enabled | up    | 2024-02-12T12:35:35.000000 | -       | -               | up            |
| cinder-volume    | cinder@cinder-ceph-flash | nova | enabled | up    | 2024-02-12T12:35:41.000000 | -       | -               | up            |
| cinder-volume    | cinder@cinder-ceph-flash | nova | enabled | up    | 2024-02-12T12:35:41.000000 | -       | -               | up            |
| cinder-volume    | cinder@cinder-ceph-hdd   | nova | enabled | up    | 2024-02-12T12:35:39.000000 | -       | -               | up            |
+------------------+--------------------------+------+---------+-------+----------------------------+---------+-----------------+---------------+