Cinder

CINDER retype fails when original volume has no volume-type

Bug #1547546 reported by Tom Barron on 2016-02-19

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Cinder	Fix Released	High	Ryan McNair	Cinder mitaka-rc1 "RC1"

Bug Description

The command 'cinder retype <volume-id> <type-id> --migration-policy on-demand' fails with the following exception when the volume with <volume-id> has no volume type:

2016-02-19 09:36:35.323 INFO cinder.volume.manager [req-c4dca4bc-2e42-4a0b-8cd8-f8131380aaa3 7dff5ccdadaa4306b7a9afebae06c4cd a83bb7aae90c49d3a0e9ab64e9f968d8] Deleted volume successfully.
2016-02-19 09:36:35.323 DEBUG oslo_concurrency.lockutils [req-c4dca4bc-2e42-4a0b-8cd8-f8131380aaa3 7dff5ccdadaa4306b7a9afebae06c4cd a83bb7aae90c49d3a0e9ab64e9f968d8] Lock "01b77d51-0b27-4700-9db9-0bcc29d3b29c-delete_volume" released by "cinder.volume.manager.lvo_inner2" :: held 0.143s from (pid=31529) inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2016-02-19 09:36:35.342 ERROR oslo_messaging.rpc.dispatcher [req-c4dca4bc-2e42-4a0b-8cd8-f8131380aaa3 7dff5ccdadaa4306b7a9afebae06c4cd a83bb7aae90c49d3a0e9ab64e9f968d8] Exception during message handling: Instance <VolumeTypes at 0x7ffa5d8fb250> has a NULL identity key. If this is an auto-generated value, check that the database table allows generation of new primary key values, and that the mapped Column object is configured to expect these generated values. Ensure also that this flush() is not occurring at an inappropriate time, such aswithin a load() event.
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 143, in _dispatch_and_reply
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 189, in _dispatch
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 2246, in retype
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher new_reservations, status_update)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher self.force_reraise()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 2242, in retype
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher new_type_id=new_type_id)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1900, in migrate_volume
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher volume.save()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher self.force_reraise()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1893, in migrate_volume
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher new_type_id)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1708, in _migrate_volume_generic
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher new_volume)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher self.force_reraise()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1693, in _migrate_volume_generic
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher error=False)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 117, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/volume/manager.py", line 1810, in migrate_volume_completion
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher updated_new = volume.finish_volume_migration(new_volume)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/objects/volume.py", line 409, in finish_volume_migration
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher dest_volume.save()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 223, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return fn(self, *args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/objects/volume.py", line 312, in save
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher db.volume_update(self._context, self.id, updates)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/db/api.py", line 225, in volume_update
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return IMPL.volume_update(context, volume_id, values)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 192, in wrapper
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 1709, in volume_update
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher return volume_ref
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 502, in __exit__
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher self.rollback()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 499, in __exit__
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher self.commit()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in commit
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher self._prepare_impl()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in _prepare_impl
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher self.session.flush()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2027, in flush
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher self._flush(objects)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2145, in _flush
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher transaction.rollback(_capture_exception=True)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2115, in _flush
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher flush_context.finalize_flush_changes()
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 395, in finalize_flush_changes
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher self.session._register_newly_persistent(other)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 1535, in _register_newly_persistent
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher % state_str(state)
2016-02-19 09:36:35.342 TRACE oslo_messaging.rpc.dispatcher FlushError: Instance <VolumeTypes at 0x7ffa5d8fb250> has a NULL identity key. If this is an auto-generated value, check that the database table allows generation of new primary key values, and that the mapped Column object is configured to expect these generated values. Ensure also that this flush() is not occurring at an inappropriate time, such aswithin a load() event.

Tags:

Revision history for this message

Tom Barron (tpb) wrote on 2016-02-19:

This is not a duplicate of [1]. There fixing a similar retype-failure required changes to volume_types_diff() to handle the case where the original volume had no volume type. Here we have an exception bubbling up from sqlaclhemy with a complaint that "VolumeTypes at 0x7ffa5d8fb250> has a NULL identity key."

[1] https://bugs.launchpad.net/cinder/+bug/1311058

Revision history for this message

Tom Barron (tpb) wrote on 2016-02-19:

Download full text (7.0 KiB)

Steps to reproduce:

Use two backends that have not implemented driver retype() so that the fallback migrate path will be taken.

1) create a volume in one of the backends without a volume-type (not specified in the command, and no default_volume_type set in cinder.conf).

$ cinder create 1
...
$ cinder show 6eb80ff1-------- | Property +------------- | attachments | availability_zone | bootable | consistencygroup_id | created_at | description | encrypted | id | metadata | migration_status | multiattach | name | os-vol- +-------------b55e-4e29-a8c6-7e2f8a64aa8f
/>--------------------------+--------------------------------------------+
| Value |
/>--------------------------+--------------------------------------------+
| [] |
| nova |
| false |
| None |
| 2016-02-19T14:33:40.000000 |
| None |
| False |
| 6eb80ff1-b55e-4e29-a8c6-7e2f8a64aa8f |
| {} |
| None |
| False |
| None |
/>host-attr:host | devstack-vm-master@nfsdriver-2#nfsdriver-2 |
/>mig-status-attr:migstat | None |
/>mig-status-attr:name_id | None |
/>tenant-attr:tenant_id | a83bb7aae90c49d3a0e9ab64e9f968d8 |
/>replication:driver_data | None |
/>replication:extended_status | None |
| disabled |
| 1 |
| None |
| None |
| available |
| 2016-02-19T14:33:41.000000 |
| 7dff5ccdadaa4306b7a9afebae06c4cd |
| None |
/>--------------------------+--------------------------------------------+

Note that this volume was placed on host nfsdriver-2#nfsdriver-2 but has no volume_type.

2) create volume type with the other backend specified as value for volume_backend_name key:

$ cinder type-create nfsdriver-1
+-----------------------...

Steps to reproduce:

Use two backends that have not implemented driver retype() so that the fallback migrate path will be taken.

1) create a volume in one of the backends without a volume-type (not specified in the command, and no default_volume_type set in cinder.conf).

$ cinder create 1
...
$ cinder show 6eb80ff1-b55e-4e29-a8c6-7e2f8a64aa8f
+---------------------------------------+--------------------------------------------+
|                Property               |                   Value                    |
+---------------------------------------+--------------------------------------------+
|              attachments              |                     []                     |
|           availability_zone           |                    nova                    |
|                bootable               |                   false                    |
|          consistencygroup_id          |                    None                    |
|               created_at              |         2016-02-19T14:33:40.000000         |
|              description              |                    None                    |
|               encrypted               |                   False                    |
|                   id                  |    6eb80ff1-b55e-4e29-a8c6-7e2f8a64aa8f    |
|                metadata               |                     {}                     |
|            migration_status           |                    None                    |
|              multiattach              |                   False                    |
|                  name                 |                    None                    |
|         os-vol-host-attr:host         | devstack-vm-master@nfsdriver-2#nfsdriver-2 |
|     os-vol-mig-status-attr:migstat    |                    None                    |
|     os-vol-mig-status-attr:name_id    |                    None                    |
|      os-vol-tenant-attr:tenant_id     |      a83bb7aae90c49d3a0e9ab64e9f968d8      |
|   os-volume-replication:driver_data   |                    None                    |
| os-volume-replication:extended_status |                    None                    |
|           replication_status          |                  disabled                  |
|                  size                 |                     1                      |
|              snapshot_id              |                    None                    |
|              source_volid             |                    None                    |
|                 status                |                 available                  |
|               updated_at              |         2016-02-19T14:33:41.000000         |
|                user_id                |      7dff5ccdadaa4306b7a9afebae06c4cd      |
|              volume_type              |                    None                    |
+---------------------------------------+--------------------------------------------+

Note that this volume was placed on host nfsdriver-2#nfsdriver-2 but has no volume_type.

2) create volume type with the other backend specified as value for  volume_backend_name key:

$ cinder type-key nfsdriver-1 set volume_backend_name=nfsdriver-2

3) run the retype command to retype the volume to the new volume type:

$ cinder retype 6eb80ff1-b55e-4e29-a8c6-7e2f8a64aa8f 58131341-4cc9-412b-ab10-1fe87e8b18cc --migration-policy on-demand
...

The retype fails and the volume is left with migration status error:

$ cinder show 6eb80ff1-b55e-4e29-a8c6-7e2f8a64aa8f
+---------------------------------------+--------------------------------------------+
|                Property               |                   Value                    |
+---------------------------------------+--------------------------------------------+
|              attachments              |                     []                     |
|           availability_zone           |                    nova                    |
|                bootable               |                   false                    |
|          consistencygroup_id          |                    None                    |
|               created_at              |         2016-02-19T14:33:40.000000         |
|              description              |                    None                    |
|               encrypted               |                   False                    |
|                   id                  |    6eb80ff1-b55e-4e29-a8c6-7e2f8a64aa8f    |
|                metadata               |                     {}                     |
|            migration_status           |                   error                    |
|              multiattach              |                   False                    |
|                  name                 |                    None                    |
|         os-vol-host-attr:host         | devstack-vm-master@nfsdriver-2#nfsdriver-2 |
|     os-vol-mig-status-attr:migstat    |                   error                    |
|     os-vol-mig-status-attr:name_id    |                    None                    |
|      os-vol-tenant-attr:tenant_id     |      a83bb7aae90c49d3a0e9ab64e9f968d8      |
|   os-volume-replication:driver_data   |                    None                    |
| os-volume-replication:extended_status |                    None                    |
|           replication_status          |                  disabled                  |
|                  size                 |                     1                      |
|              snapshot_id              |                    None                    |
|              source_volid             |                    None                    |
|                 status                |                 available                  |
|               updated_at              |         2016-02-19T14:36:35.000000         |
|                user_id                |      7dff5ccdadaa4306b7a9afebae06c4cd      |
|              volume_type              |                    None                    |
+---------------------------------------+--------------------------------------------+

Revision history for this message

Eric Harney (eharney) wrote on 2016-02-19:

If nothing else, this should not throw errors from the SQLAlchemy layer.

tags:	added: retype
Changed in cinder:
importance:	Undecided → High

Tom Barron (tpb) on 2016-02-19

Changed in cinder:
assignee:	nobody → Tom Barron (tpb)
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-02-20: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/282579

Sean McGinnis (sean-mcginnis) on 2016-03-07

Changed in cinder:
milestone:	none → mitaka-rc1

OpenStack Infra (hudson-openstack) on 2016-03-14

Changed in cinder:
assignee:	Tom Barron (tpb) → Ryan McNair (rdmcnair)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-15: Fix merged to cinder (master)

Reviewed: https://review.openstack.org/282579
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=492b350a3fec957263430a1e67457a7386254c61
Submitter: Jenkins
Branch: master

commit 492b350a3fec957263430a1e67457a7386254c61
Author: Tom Barron <email address hidden>
Date: Fri Feb 19 18:19:14 2016 -0500

Fix retype failure when original has no volume type

    cinder.objects.volume.Volume.finish_volume_migration()
    should skip the volume_type_id field when swapping fields
    between "source" and "dest" volumes. The "dest" volume
    has already been created with appropriate volume_type_id
    and the "source" may have not have a volume type.

    This commit also changes a LOG.error() to a LOG.exception()
    in where finish_volume_migration is called in order to
    show the exception traceback in its calling context.

Change-Id: I2caf4d3f4aa088d099548e6e88d1776b4cc5810c
Closes-bug: #1547546

Changed in cinder:
status:	In Progress → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.