In our case database was inconsistent eg. 'service_uuid' field was sometimes NULL:
MariaDB [cinder]> select id,volume_type_id,status,service_uuid from volumes where host like '%ik03@rbd-ec%' and status!='deleted' and service_uuid is NULL;
Our workaround/fix was:
MariaDB [cinder]> update volumes set service_uuid='81d03e44-21ab-41ea-81b6-a3b5aac8a3b6' where host like '%ik03@rbd-ec%' and status!='deleted' and service_uuid is NULL;
We upgraded openstack-ansible from rocky to stain and cinder-manage also fails (see below to fix/workaround).
~# cinder-manage --debug db online_ data_migrations --max_count 1 service_ uuids_online_ data_migration ------- ------- ------- ------- ------- ---+--- ------- ----+-- ------- --+ ------- ------- ------- ------- ------- ---+--- ------- ----+-- ------- --+ specs_online_ data_migration | 0 | 0 | service_ online_ migration | 0 | 0 | uuids_online_ data_migration | 0 | 0 | targets_ online_ data_migration | 0 | 0 | service_ uuids_online_ data_migration | 0 | 0 | ------- ------- ------- ------- ------- ---+--- ------- ----+-- ------- --+
; echo $?
Error attempting to run volume_
+------
| Migration | Total Needed | Completed |
+------
| attachment_
| backup_
| service_
| shared_
| volume_
+------
Some migrations failed unexpectedly. Check log for details.
2
~# journalctl --since "1 hour ago" | grep cinder.cmd.manage | grep Key api-container- ebba9597 cinder-manage[326]: 2019-08-07 13:59:26.692 326 ERROR cinder.cmd.manage [req-d0a34065- a288-4b2b- 9fad-1b915c1f4e ea - - - - -] Error attempting to run volume_ service_ uuids_online_ data_migration: KeyError: u'ik03@rbd-ec'
Aug 07 13:59:26 ik01-cinder-
In our case database was inconsistent eg. 'service_uuid' field was sometimes NULL: type_id, status, service_ uuid from volumes where host like '%ik03@rbd-ec%' and status!='deleted' and service_uuid is NULL;
MariaDB [cinder]> select id,volume_
Our workaround/fix was: uuid='81d03e44- 21ab-41ea- 81b6-a3b5aac8a3 b6' where host like '%ik03@rbd-ec%' and status!='deleted' and service_uuid is NULL;
MariaDB [cinder]> update volumes set service_
~# /openstack/ venvs/cinder- 19.0.0/ bin/cinder- manage --debug db online_ data_migrations ; echo $? service_ uuids_online_ data_migration, 49 migrated ------- ------- ------- ------- ------- ---+--- ------- ----+-- ------- --+ ------- ------- ------- ------- ------- ---+--- ------- ----+-- ------- --+ specs_online_ data_migration | 0 | 0 | service_ online_ migration | 0 | 0 | uuids_online_ data_migration | 0 | 0 | targets_ online_ data_migration | 0 | 0 | service_ uuids_online_ data_migration | 49 | 49 | ------- ------- ------- ------- ------- ---+--- ------- ----+-- ------- --+
Running batches of 50 until complete.
49 rows matched query volume_
+------
| Migration | Total Needed | Completed |
+------
| attachment_
| backup_
| service_
| shared_
| volume_
+------
0
Hope it helps someone to save few hours.