workers table has many volumes stuck with null service_id being NULL
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
New
|
Undecided
|
Unassigned |
Bug Description
If cinder restarts or rabbitmq restarts/bounces before the volume is scheduled on a host a volume can be stuck in 'creating' and the workers table is left with an entry without a service_id. This makes the volume not cleanable and will never get out of creating status even on restarts.
The following is from our QA system where we redeploy cinder pods on kubernetes frequently to update
rabbitmq or cinder services.
These entries will NEVER get cleaned as the volume service requires the service_id to be set in order for the db query to find them at do_cleanup() time in the volume manager.
https:/
MariaDB root@127.
+------
| count(*) | status |
+------
| 1951 | creating |
| 211 | deleting |
| 1 | OK |
+------
This is one of our production systems.
MariaDB root@127.
+------
| count(*) | status |
+------
| 292 | creating |
| 42 | deleting |
| 1 | OK |
+------