online volume migration fails due to db schema change

Bug #1219097 reported by Bill Owen on 2013-08-30
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
High
Avishay Traeger

Bug Description

After installing latest cinder code and applying db sync operation, online volume migration now fails with the following traceback:

2013-08-30 14:26:43.411 1399033 ERROR cinder.openstack.common.rpc.amqp [req-78555fa4-c79c-4160-993d-05d60b001222 d8a70c91de4d41cc9bf994c8d23e4479 af64382b5b1449189608c0a3c9c39a0c] Exception during message handling
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 433, in _process_data
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp **args)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 686, in migrate_volume_completion
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp self.db.finish_volume_migration(ctxt, volume_id, new_volume_id)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/db/api.py", line 198, in finish_volume_migration
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp return IMPL.finish_volume_migration(context, src_vol_id, dest_vol_id)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.py", line 120, in wrapper
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp return f(*args, **kwargs)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.py", line 1121, in finish_volume_migration
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp filter_by(id=dest_vol_id).\
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2679, in delete
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp result = session.execute(delete_stmt, params=self._params)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/db/sqlalchemy/session.py", line 485, in _wrap
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp raise exception.DBError(e)
2013-08-30 14:26:43.411 1399033 TRACE cinder.openstack.common.rpc.amqp DBError: (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`cinder`.`volume_admin_metadata`, CONSTRAINT `volume_admin_metadata_ibfk_1` FOREIGN KEY (`volume_id`) REFERENCES `volumes` (`id`))') 'DELETE FROM volumes WHERE volumes.id = %s' ('86475de8-daba-45e3-9c15-76ca0941efb6',)

There is new table in cinder db volume_admin_metadata with FK relationship with volumes. This admin metadata needs to be handled in the same way as normal volume metadata.

Changed in cinder:
assignee: nobody → Avishay Traeger (avishay-il)
Avishay Traeger (avishay-il) wrote :

So the problem here is that admin metadata is automatically created for every volume with admin context, which includes the migration destination volume. Regular metadata shouldn't exist for the new volume, but will check for that too.

Changed in cinder:
status: New → Confirmed
importance: Undecided → High
Avishay Traeger (avishay-il) wrote :

While this exception SHOULD be raised, it is not raised by my unit tests. Which DB are you using? Running standard devstack or something else?

Fix proposed to branch: master
Review: https://review.openstack.org/44794

Changed in cinder:
status: Confirmed → In Progress
Bill Owen (billowen) wrote :

Hi Avishay
I am using mysql. The problem wasn't seen in unit tests, but in exercising migration in test environment.

Avishay Traeger (avishay-il) wrote :

From my understanding, mysql with InnoDB should check, and from what I could tell my devstack environment is using that. I wasn't able to reproduce it though. I created a new volume, attached it to a VM, and did host-based volume migration successfully. No errors in the logs. Will keep digging...

Changed in cinder:
milestone: none → havana-rc1

Reviewed: https://review.openstack.org/44794
Committed: http://github.com/openstack/cinder/commit/d2479f1e88e1eff777afbc922ae1da63afcc7716
Submitter: Jenkins
Branch: master

commit d2479f1e88e1eff777afbc922ae1da63afcc7716
Author: Avishay Traeger <email address hidden>
Date: Mon Sep 2 20:23:45 2013 +0300

    Soft delete tmp migration volume

    Previously the row for the temporary migration volume was deleted from
    the database, which caused a foreign key violation for its admin
    metadata. Regardless, the volume should have been soft deleted from the
    database anyway, and this patch takes care of that.

    Change-Id: I8e4942b0e21ec79c90aa95b18b5612b5b133fd62
    Closes-Bug: #1219097

Changed in cinder:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2013-10-04
Changed in cinder:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2013-10-17
Changed in cinder:
milestone: havana-rc1 → 2013.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers