online volume migration fails due to db schema change

[Bill Owen]

After installing latest cinder code and applying db sync operation, online volume migration now fails with the following traceback:

2013-08-30 14:26:43.411 1399033 ERROR cinder.openstack.common.rpc.amqp [req-78555fa4-c79c-4160-993d-05d60b001222 d8a70c91de4d41cc9bf994c8d23e4479 af64382b5b1449189608c0a3c9c39a0c] Exception during message handling
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp **args)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 686, in migrate_volume_completion
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp self.db.finish_volume_migration(ctxt, volume_id, new_volume_id)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/db/api.py", line 198, in finish_volume_migration
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp return IMPL.finish_volume_migration(context, src_vol_id, dest_vol_id)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.py", line 120, in wrapper
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp return f(*args, **kwargs)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.py", line 1121, in finish_volume_migration
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp filter_by(id=dest_vol_id).\
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2679, in delete
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp result = session.execute(delete_stmt, params=self._params)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/db/sqlalchemy/session.py", line 485, in _wrap
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp raise exception.DBError(e)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp DBError: (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_admin_metadata`, CONSTRAINT `volume_admin_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))') 'DELETE FROM volumes WHERE volumes.id = %s' ('86475de8-daba-45e3-9c15-76ca0941efb6',)

There is new table in cinder db volume_admin_metadata with FK relationship with volumes. This admin metadata needs to be handled in the same way as normal volume metadata.

[Avishay Traeger (avishay-il)]

So the problem here is that admin metadata is automatically created for every volume with admin context, which includes the migration destination volume. Regular metadata shouldn't exist for the new volume, but will check for that too.

[Avishay Traeger (avishay-il)]

While this exception SHOULD be raised, it is not raised by my unit tests. Which DB are you using? Running standard devstack or something else?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/44794

[Bill Owen (billowen)]

Hi Avishay
I am using mysql. The problem wasn't seen in unit tests, but in exercising migration in test environment.

[Avishay Traeger (avishay-il)]

From my understanding, mysql with InnoDB should check, and from what I could tell my devstack environment is using that. I wasn't able to reproduce it though. I created a new volume, attached it to a VM, and did host-based volume migration successfully. No errors in the logs. Will keep digging...

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/44794
Committed: http://github.com/openstack/cinder/commit/d2479f1e88e1eff777afbc922ae1da63afcc7716
Submitter: Jenkins
Branch: master

commit d2479f1e88e1eff777afbc922ae1da63afcc7716
Author: Avishay Traeger <email address hidden>
Date: Mon Sep 2 20:23:45 2013 +0300

    Soft delete tmp migration volume

    Previously the row for the temporary migration volume was deleted from
    the database, which caused a foreign key violation for its admin
    metadata. Regardless, the volume should have been soft deleted from the
    database anyway, and this patch takes care of that.

    Change-Id: I8e4942b0e21ec79c90aa95b18b5612b5b133fd62
    Closes-Bug: #1219097