"cinder-manage db purge" fails if live volumes refer to deleted services
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
New
|
Undecided
|
Unassigned | ||
OpenStack Cinder Charm |
Incomplete
|
Undecided
|
Unassigned |
Bug Description
I'm not sure if this is an issue that typically happens in cinder by itself; it may be related to how the cinder Juju charm manages things. However, I observed this today.
During a run of "sudo cinder-manage db purge 90", I get this error:
2024-05-21 22:57:00.325 479170 INFO cinder.
2024-05-21 22:57:00.328 479170 ERROR cinder.
[SQL: DELETE FROM services WHERE services.deleted IS true AND services.deleted_at < %(deleted_at_1)s]
[parameters: {'deleted_at_1': datetime.
(Background on this error at: https:/
Purge command failed, check cinder-manage logs for more details.
Indeed, it seems that the services should *not* be deleted, as I still have live volumes which refer to the deleted services:
ysql> SELECT Count(*) FROM cinder.volumes v LEFT JOIN cinder.services s ON v.service_uuid = s.uuid WHERE v.deleted=0 AND s.deleted IS TRUE;
+----------+
| Count(*) |
+----------+
| 899 |
+----------+
1 row in set (0.00 sec)
We sometimes have a case where we end up with extra cinder services showing up in "openstack volume service list", and this is typically "cleaned up" via running the Juju action "juju run-action --wait cinder/leader remove-services". However, I suspect that this "cleanup" action causes the stale volumes. As a side effect, while the "cinder-manage db purge" seems to at least partially run, I am not left with confidence that the cleanup has run on all the tables that it could.
As I write this, I suspect this is most likely a bug for charm-cinder rather than cinder proper, but I'll file it under both for the sake of review.
Cinder version in use: 2:20.3.
OS: Ubuntu 22.04 Jammy
Cinder charm in use: yoga/stable channel, revision 656
Changed in charm-cinder: | |
status: | New → Incomplete |
@Paul - can you provide some more commentary about the additional cinder services that show up and the circumstances around this?