volume creation based on image source fails when using ceph backend

Bug #1580222 reported by Greg Faust on 2016-05-10
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
openstack-ansible
High
Flávio Ramalho

Bug Description

when using ceph as a backend for cinder I am running into an issue where volume creation based on an image source fails.

empty volume creation works fine, it's only volume creation based on an image source that fails.

Here is the relevant excerpt from /var/log/cinder/cinder-volume.log (within the cinder-volume container):

2016-05-10 14:38:30.636 10357 INFO cinder.volume.flows.manager.create_volume [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Volume 2ffb89d3-4048-4927-8795-4d13c825cf84: being created as image with specification: {'status': u'creating', 'image_location': (u'rbd://1a99be1b-fd46-4b67-b2f1-3ea3efdd000f/glance/28bf4043-d449-416a-a994-234aa613ed7c/snap', None), 'volume_size': 20, 'volume_name': 'volume-2ffb89d3-4048-4927-8795-4d13c825cf84', 'image_id': u'28bf4043-d449-416a-a994-234aa613ed7c', 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f648f89a890>, 'image_meta': {u'status': u'active', u'virtual_size': None, u'name': u'trusty64', u'tags': [], u'container_format': u'bare', u'created_at': datetime.datetime(2016, 5, 5, 21, 47, 13, tzinfo=<iso8601.Utc>), u'disk_format': u'qcow2', u'updated_at': datetime.datetime(2016, 5, 5, 21, 47, 20, tzinfo=<iso8601.Utc>), u'visibility': u'private', 'properties': {}, u'owner': u'a0adde92c1b74a67ad8c640168c28632', u'protected': False, u'id': u'28bf4043-d449-416a-a994-234aa613ed7c', u'file': u'/v2/images/28bf4043-d449-416a-a994-234aa613ed7c/file', u'checksum': u'e05b2f067604a1b4d0e9aa430b4e8721', u'min_disk': 0, u'direct_url': u'rbd://1a99be1b-fd46-4b67-b2f1-3ea3efdd000f/glance/28bf4043-d449-416a-a994-234aa613ed7c/snap', u'min_ram': 0, u'size': 259457536}}
2016-05-10 14:38:33.648 10357 INFO cinder.image.image_utils [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Image download 247.00 MB at 82.89 MB/s
2016-05-10 14:38:33.769 10357 WARNING cinder.volume.manager [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1e8040cc-80f2-40a4-ad4f-a15ab329ba35) transitioned into state 'FAILURE' from state 'RUNNING'
5 predecessors (most recent first):
  Atom 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume_ref': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,consistencygroup=<?>,consistencygroup_id=None,created_at=2016-05-10T14:38:25Z,deleted=False,deleted_at=None,display_description='',display_name='test2',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,host='hostname-cinder-volumes-container-16ac4806@rbd#rbd',id=2ffb89d3-4048-4927-8795-4d13c825cf84,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id=a0adde92c1b74a67ad8c640168c28632,provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status='disabled',scheduled_at=2016-05-10T14:38:29Z,size=20,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2016-05-10T14:38:30Z,user_id=b8e11a21c5f84dd2b629d480c00b9953,volume_attachment=<?>,volume_type=VolumeType(3aa4c5f4-720b-4566-af2e-71858c355504),volume_type_id=3aa4c5f4-720b-4566-af2e-71858c355504), 'context': <cinder.context.RequestContext object at 0x7f6493d02d10>}, 'provides': None}
  |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume_ref': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,consistencygroup=<?>,consistencygroup_id=None,created_at=2016-05-10T14:38:25Z,deleted=False,deleted_at=None,display_description='',display_name='test2',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,host='hostname-cinder-volumes-container-16ac4806@rbd#rbd',id=2ffb89d3-4048-4927-8795-4d13c825cf84,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id=a0adde92c1b74a67ad8c640168c28632,provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status='disabled',scheduled_at=2016-05-10T14:38:29Z,size=20,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2016-05-10T14:38:30Z,user_id=b8e11a21c5f84dd2b629d480c00b9953,volume_attachment=<?>,volume_type=VolumeType(3aa4c5f4-720b-4566-af2e-71858c355504),volume_type_id=3aa4c5f4-720b-4566-af2e-71858c355504), 'request_spec': {u'source_replicaid': None, u'volume_properties': {u'status': u'creating', u'volume_type_id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'snapshot_id': None, u'display_name': u'test2', u'availability_zone': u'nova', u'reservations': [u'296cb2cb-3aad-4bee-87ad-8d0faf1b0ce8', u'8da597f1-adad-4ca5-b63a-d7e513ab4834', u'e29c90da-a941-4ab5-9dff-a420005ba86d', u'2ac1317f-f6fc-4948-8358-54d3db86503b'], u'multiattach': False, u'attach_status': u'detached', u'display_description': u'', u'cgsnapshot_id': None, u'qos_specs': None, u'encryption_key_id': None, u'replication_status': u'disabled', u'source_replicaid': None, u'source_volid': None, u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'project_id': u'a0adde92c1b74a67ad8c640168c28632', u'consistencygroup_id': None, u'size': 20, u'metadata': {}}, u'source_volid': None, u'cgsnapshot_id': None, u'volume': {u'migration_status': None, u'provider_id': None, u'availability_zone': u'nova', u'terminated_at': None, u'updated_at': None, u'provider_geometry': None, u'ec2_id': None, u'replication_status': u'disabled', u'snapshot_id': None, u'replication_extended_status': None, u'deleted_at': None, u'id': u'2ffb89d3-4048-4927-8795-4d13c825cf84', u'size': 20, u'display_name': u'test2', u'display_description': u'', u'name_id': u'2ffb89d3-4048-4927-8795-4d13c825cf84', u'volume_admin_metadata': [], u'project_id': u'a0adde92c1b74a67ad8c640168c28632', u'launched_at': None, u'metadata': {}, u'status': u'creating', u'volume_type_id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'multiattach': False, u'deleted': False, u'provider_location': None, u'volume_glance_metadata': [], u'host': None, u'consistencygroup_id': None, u'source_volid': None, u'provider_auth': None, u'previous_status': None, u'name': u'volume-2ffb89d3-4048-4927-8795-4d13c825cf84', u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'bootable': False, u'created_at': u'2016-05-10T14:38:25.231113', u'attach_status': u'detached', u'volume_metadata': [], u'_name_id': None, u'encryption_key_id': None, u'replication_driver_data': None, u'scheduled_at': None}, u'image_id': u'28bf4043-d449-416a-a994-234aa613ed7c', u'snapshot_id': None, u'consistencygroup_id': None, u'volume_type': {u'name': u'rbd', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2016-05-05T21:15:45.000000', u'updated_at': None, u'extra_specs': {u'volume_backend_name': u'rbd'}, u'is_public': True, u'deleted_at': None, u'id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'description': None}, u'volume_id': u'2ffb89d3-4048-4927-8795-4d13c825cf84', u'resource_properties': {u'status': u'creating', u'volume_type_id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'snapshot_id': None, u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'availability_zone': u'nova', u'reservations': [u'296cb2cb-3aad-4bee-87ad-8d0faf1b0ce8', u'8da597f1-adad-4ca5-b63a-d7e513ab4834', u'e29c90da-a941-4ab5-9dff-a420005ba86d', u'2ac1317f-f6fc-4948-8358-54d3db86503b'], u'multiattach': False, u'attach_status': u'detached', u'display_description': u'', u'cgsnapshot_id': None, u'qos_specs': None, u'encryption_key_id': None, u'replication_status': u'disabled', u'source_replicaid': None, u'source_volid': None, u'display_name': u'test2', u'project_id': u'a0adde92c1b74a67ad8c640168c28632', u'consistencygroup_id': None, u'size': 20, u'metadata': {}}}, 'context': <cinder.context.RequestContext object at 0x7f6493d02d10>}, 'provides': {'status': u'creating', 'image_location': (u'rbd://1a99be1b-fd46-4b67-b2f1-3ea3efdd000f/glance/28bf4043-d449-416a-a994-234aa613ed7c/snap', None), 'image_id': u'28bf4043-d449-416a-a994-234aa613ed7c', 'volume_id': '2ffb89d3-4048-4927-8795-4d13c825cf84', 'volume_size': 20, 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f648f89a890>, 'image_meta': {u'status': u'active', u'virtual_size': None, u'name': u'trusty64', u'tags': [], u'container_format': u'bare', u'created_at': datetime.datetime(2016, 5, 5, 21, 47, 13, tzinfo=<iso8601.Utc>), u'disk_format': u'qcow2', u'updated_at': datetime.datetime(2016, 5, 5, 21, 47, 20, tzinfo=<iso8601.Utc>), u'visibility': u'private', 'properties': {}, u'owner': u'a0adde92c1b74a67ad8c640168c28632', u'protected': False, u'id': u'28bf4043-d449-416a-a994-234aa613ed7c', u'file': u'/v2/images/28bf4043-d449-416a-a994-234aa613ed7c/file', u'checksum': u'e05b2f067604a1b4d0e9aa430b4e8721', u'min_disk': 0, u'direct_url': u'rbd://1a99be1b-fd46-4b67-b2f1-3ea3efdd000f/glance/28bf4043-d449-416a-a994-234aa613ed7c/snap', u'min_ram': 0, u'size': 259457536}, 'volume_name': 'volume-2ffb89d3-4048-4927-8795-4d13c825cf84', 'type': 'image'}}
     |__Atom 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'volume_ref': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,consistencygroup=<?>,consistencygroup_id=None,created_at=2016-05-10T14:38:25Z,deleted=False,deleted_at=None,display_description='',display_name='test2',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,host='hostname-cinder-volumes-container-16ac4806@rbd#rbd',id=2ffb89d3-4048-4927-8795-4d13c825cf84,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id=a0adde92c1b74a67ad8c640168c28632,provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status='disabled',scheduled_at=2016-05-10T14:38:29Z,size=20,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2016-05-10T14:38:30Z,user_id=b8e11a21c5f84dd2b629d480c00b9953,volume_attachment=<?>,volume_type=VolumeType(3aa4c5f4-720b-4566-af2e-71858c355504),volume_type_id=3aa4c5f4-720b-4566-af2e-71858c355504), 'filter_properties': {u'config_options': {}, u'request_spec': {u'volume_id': u'2ffb89d3-4048-4927-8795-4d13c825cf84', u'source_replicaid': None, u'volume_properties': {u'status': u'creating', u'volume_type_id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'display_name': u'test2', u'multiattach': False, u'reservations': [u'296cb2cb-3aad-4bee-87ad-8d0faf1b0ce8', u'8da597f1-adad-4ca5-b63a-d7e513ab4834', u'e29c90da-a941-4ab5-9dff-a420005ba86d', u'2ac1317f-f6fc-4948-8358-54d3db86503b'], u'availability_zone': u'nova', u'attach_status': u'detached', u'display_description': u'', u'cgsnapshot_id': None, u'source_replicaid': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'replication_status': u'disabled', u'snapshot_id': None, u'source_volid': None, u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'project_id': u'a0adde92c1b74a67ad8c640168c28632', u'consistencygroup_id': None, u'size': 20}, u'source_volid': None, u'cgsnapshot_id': None, u'volume': {u'migration_status': None, u'provider_id': None, u'availability_zone': u'nova', u'terminated_at': None, u'updated_at': None, u'provider_geometry': None, u'ec2_id': None, u'replication_status': u'disabled', u'snapshot_id': None, u'replication_extended_status': None, u'deleted_at': None, u'id': u'2ffb89d3-4048-4927-8795-4d13c825cf84', u'size': 20, u'display_name': u'test2', u'display_description': u'', u'name_id': u'2ffb89d3-4048-4927-8795-4d13c825cf84', u'volume_admin_metadata': [], u'encryption_key_id': None, u'project_id': u'a0adde92c1b74a67ad8c640168c28632', u'launched_at': None, u'metadata': {}, u'status': u'creating', u'volume_type_id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'multiattach': False, u'deleted': False, u'provider_location': None, u'volume_glance_metadata': [], u'host': None, u'consistencygroup_id': None, u'source_volid': None, u'provider_auth': None, u'previous_status': None, u'name': u'volume-2ffb89d3-4048-4927-8795-4d13c825cf84', u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'bootable': False, u'created_at': u'2016-05-10T14:38:25.231113', u'attach_status': u'detached', u'_name_id': None, u'volume_metadata': [], u'replication_driver_data': None, u'scheduled_at': None}, u'image_id': u'28bf4043-d449-416a-a994-234aa613ed7c', u'consistencygroup_id': None, u'volume_type': {u'name': u'rbd', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2016-05-05T21:15:45.000000', u'updated_at': None, u'extra_specs': {u'volume_backend_name': u'rbd'}, u'is_public': True, u'deleted_at': None, u'id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'description': None}, u'snapshot_id': None, u'resource_properties': {u'status': u'creating', u'volume_type_id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'multiattach': False, u'reservations': [u'296cb2cb-3aad-4bee-87ad-8d0faf1b0ce8', u'8da597f1-adad-4ca5-b63a-d7e513ab4834', u'e29c90da-a941-4ab5-9dff-a420005ba86d', u'2ac1317f-f6fc-4948-8358-54d3db86503b'], u'availability_zone': u'nova', u'attach_status': u'detached', u'display_description': u'', u'cgsnapshot_id': None, u'source_replicaid': None, u'metadata': {}, u'qos_specs': None, u'encryption_key_id': None, u'replication_status': u'disabled', u'snapshot_id': None, u'source_volid': None, u'display_name': u'test2', u'project_id': u'a0adde92c1b74a67ad8c640168c28632', u'consistencygroup_id': None, u'size': 20}}, u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'availability_zone': u'nova', u'volume_type': {u'name': u'rbd', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2016-05-05T21:15:45.000000', u'updated_at': None, u'extra_specs': {u'volume_backend_name': u'rbd'}, u'is_public': True, u'deleted_at': None, u'id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'description': None}, u'qos_specs': None, u'retry': {u'num_attempts': 2, u'hosts': [u'hostname@rbd#rbd', u'hostname-cinder-volumes-container-16ac4806@rbd#rbd']}, u'metadata': {}, u'resource_type': {u'name': u'rbd', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2016-05-05T21:15:45.000000', u'updated_at': None, u'extra_specs': {u'volume_backend_name': u'rbd'}, u'is_public': True, u'deleted_at': None, u'id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'description': None}, u'size': 20}, 'context': <cinder.context.RequestContext object at 0x7f6493d02d10>, 'request_spec': {u'source_replicaid': None, u'volume_properties': {u'status': u'creating', u'volume_type_id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'snapshot_id': None, u'display_name': u'test2', u'availability_zone': u'nova', u'reservations': [u'296cb2cb-3aad-4bee-87ad-8d0faf1b0ce8', u'8da597f1-adad-4ca5-b63a-d7e513ab4834', u'e29c90da-a941-4ab5-9dff-a420005ba86d', u'2ac1317f-f6fc-4948-8358-54d3db86503b'], u'multiattach': False, u'attach_status': u'detached', u'display_description': u'', u'cgsnapshot_id': None, u'qos_specs': None, u'encryption_key_id': None, u'replication_status': u'disabled', u'source_replicaid': None, u'source_volid': None, u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'project_id': u'a0adde92c1b74a67ad8c640168c28632', u'consistencygroup_id': None, u'size': 20, u'metadata': {}}, u'source_volid': None, u'cgsnapshot_id': None, u'volume': {u'migration_status': None, u'provider_id': None, u'availability_zone': u'nova', u'terminated_at': None, u'updated_at': None, u'provider_geometry': None, u'ec2_id': None, u'replication_status': u'disabled', u'snapshot_id': None, u'replication_extended_status': None, u'deleted_at': None, u'id': u'2ffb89d3-4048-4927-8795-4d13c825cf84', u'size': 20, u'display_name': u'test2', u'display_description': u'', u'name_id': u'2ffb89d3-4048-4927-8795-4d13c825cf84', u'volume_admin_metadata': [], u'project_id': u'a0adde92c1b74a67ad8c640168c28632', u'launched_at': None, u'metadata': {}, u'status': u'creating', u'volume_type_id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'multiattach': False, u'deleted': False, u'provider_location': None, u'volume_glance_metadata': [], u'host': None, u'consistencygroup_id': None, u'source_volid': None, u'provider_auth': None, u'previous_status': None, u'name': u'volume-2ffb89d3-4048-4927-8795-4d13c825cf84', u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'bootable': False, u'created_at': u'2016-05-10T14:38:25.231113', u'attach_status': u'detached', u'volume_metadata': [], u'_name_id': None, u'encryption_key_id': None, u'replication_driver_data': None, u'scheduled_at': None}, u'image_id': u'28bf4043-d449-416a-a994-234aa613ed7c', u'snapshot_id': None, u'consistencygroup_id': None, u'volume_type': {u'name': u'rbd', u'qos_specs_id': None, u'deleted': False, u'created_at': u'2016-05-05T21:15:45.000000', u'updated_at': None, u'extra_specs': {u'volume_backend_name': u'rbd'}, u'is_public': True, u'deleted_at': None, u'id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'description': None}, u'volume_id': u'2ffb89d3-4048-4927-8795-4d13c825cf84', u'resource_properties': {u'status': u'creating', u'volume_type_id': u'3aa4c5f4-720b-4566-af2e-71858c355504', u'snapshot_id': None, u'user_id': u'b8e11a21c5f84dd2b629d480c00b9953', u'availability_zone': u'nova', u'reservations': [u'296cb2cb-3aad-4bee-87ad-8d0faf1b0ce8', u'8da597f1-adad-4ca5-b63a-d7e513ab4834', u'e29c90da-a941-4ab5-9dff-a420005ba86d', u'2ac1317f-f6fc-4948-8358-54d3db86503b'], u'multiattach': False, u'attach_status': u'detached', u'display_description': u'', u'cgsnapshot_id': None, u'qos_specs': None, u'encryption_key_id': None, u'replication_status': u'disabled', u'source_replicaid': None, u'source_volid': None, u'display_name': u'test2', u'project_id': u'a0adde92c1b74a67ad8c640168c28632', u'consistencygroup_id': None, u'size': 20, u'metadata': {}}}}, 'provides': None}
        |__Atom 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'context': <cinder.context.RequestContext object at 0x7f6493d02d10>, 'volume_id': '2ffb89d3-4048-4927-8795-4d13c825cf84'}, 'provides': Volume(_name_id=None,admin_metadata={},attach_status='detached',availability_zone='nova',bootable=False,consistencygroup=<?>,consistencygroup_id=None,created_at=2016-05-10T14:38:25Z,deleted=False,deleted_at=None,display_description='',display_name='test2',ec2_id=None,encryption_key_id=None,glance_metadata=<?>,host='hostname-cinder-volumes-container-16ac4806@rbd#rbd',id=2ffb89d3-4048-4927-8795-4d13c825cf84,launched_at=None,metadata={},migration_status=None,multiattach=False,previous_status=None,project_id=a0adde92c1b74a67ad8c640168c28632,provider_auth=None,provider_geometry=None,provider_id=None,provider_location=None,replication_driver_data=None,replication_extended_status=None,replication_status='disabled',scheduled_at=2016-05-10T14:38:29Z,size=20,snapshot_id=None,snapshots=<?>,source_volid=None,status='creating',terminated_at=None,updated_at=2016-05-10T14:38:30Z,user_id=b8e11a21c5f84dd2b629d480c00b9953,volume_attachment=<?>,volume_type=VolumeType(3aa4c5f4-720b-4566-af2e-71858c355504),volume_type_id=3aa4c5f4-720b-4566-af2e-71858c355504)}
           |__Flow 'volume_create_manager'
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager Traceback (most recent call last):
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager result = task.execute(**arguments)
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 833, in execute
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager **volume_spec)
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 741, in _create_from_image
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager image_service, context, image_id) as tmp_image:
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager return self.gen.next()
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/image/image_utils.py", line 533, in fetch
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager fetch_verify_image(context, image_service, image_id, tmp)
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/image/image_utils.py", line 200, in fetch_verify_image
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager data = qemu_img_info(dest, run_as_root=run_as_root)
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/image/image_utils.py", line 63, in qemu_img_info
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager out, _err = utils.execute(*cmd, run_as_root=run_as_root)
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/utils.py", line 148, in execute
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager return processutils.execute(*cmd, **kwargs)
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 371, in execute
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager cmd=sanitized_cmd)
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager ProcessExecutionError: Unexpected error while running command.
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /var/lib/cinder/conversion/tmpUXabEL
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager Exit code: 96
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager Stdout: u''
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager Stderr: u'/openstack/venvs/cinder-13.0.1/bin/cinder-rootwrap: Executable not found: qemu-img (filter match = qemu-img)\n'
2016-05-10 14:38:33.769 10357 ERROR cinder.volume.manager
2016-05-10 14:38:33.777 10357 WARNING cinder.volume.manager [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Task 'cinder.volume.flows.manager.create_volume.CreateVolumeFromSpecTask;volume:create' (1e8040cc-80f2-40a4-ad4f-a15ab329ba35) transitioned into state 'REVERTED' from state 'REVERTING'
2016-05-10 14:38:33.780 10357 WARNING cinder.volume.manager [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Task 'cinder.volume.flows.manager.create_volume.NotifyVolumeActionTask;volume:create, create.start' (f9afed83-510a-48a5-8e7e-04cdad871558) transitioned into state 'REVERTED' from state 'REVERTING'
2016-05-10 14:38:33.783 10357 WARNING cinder.volume.manager [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeSpecTask;volume:create' (aa307174-5ff2-465e-88e3-0bc177547115) transitioned into state 'REVERTED' from state 'REVERTING'
2016-05-10 14:38:33.854 10357 WARNING cinder.volume.manager [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Task 'cinder.volume.flows.manager.create_volume.OnFailureRescheduleTask;volume:create' (536bf26f-e4a7-4445-ba92-505729069c90) transitioned into state 'REVERTED' from state 'REVERTING'
2016-05-10 14:38:33.859 10357 WARNING cinder.volume.manager [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Task 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (e7ccb801-b137-4a9a-9d2f-faa8980c2571) transitioned into state 'REVERTED' from state 'REVERTING'
2016-05-10 14:38:33.863 10357 WARNING cinder.volume.manager [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Flow 'volume_create_manager' (6493c410-18c7-46c1-9840-af701b2ff554) transitioned into state 'REVERTED' from state 'RUNNING'
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher [req-a13d56d2-ca85-4fba-a164-b1e5ce422e6f b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Exception during message handling: Unexpected error while running command.
Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /var/lib/cinder/conversion/tmpUXabEL
Exit code: 96
Stdout: u''
Stderr: u'/openstack/venvs/cinder-13.0.1/bin/cinder-rootwrap: Executable not found: qemu-img (filter match = qemu-img)\n'
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/volume/manager.py", line 630, in create_volume
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher _run_flow()
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/volume/manager.py", line 617, in _run_flow
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher flow_engine.run()
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 230, in run
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher for _state in self.run_iter(timeout=timeout):
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 308, in run_iter
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher failure.Failure.reraise_if_any(fails)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher failures[0].reraise()
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher six.reraise(*self._exc_info)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 82, in _execute_task
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher result = task.execute(**arguments)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 833, in execute
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher **volume_spec)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 741, in _create_from_image
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher image_service, context, image_id) as tmp_image:
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher return self.gen.next()
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/image/image_utils.py", line 533, in fetch
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher fetch_verify_image(context, image_service, image_id, tmp)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/image/image_utils.py", line 200, in fetch_verify_image
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher data = qemu_img_info(dest, run_as_root=run_as_root)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/image/image_utils.py", line 63, in qemu_img_info
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher out, _err = utils.execute(*cmd, run_as_root=run_as_root)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/cinder/utils.py", line 148, in execute
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher File "/openstack/venvs/cinder-13.0.1/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 371, in execute
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher cmd=sanitized_cmd)
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher Command: sudo cinder-rootwrap /etc/cinder/rootwrap.conf env LC_ALL=C qemu-img info /var/lib/cinder/conversion/tmpUXabEL
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher Exit code: 96
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher Stdout: u''
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher Stderr: u'/openstack/venvs/cinder-13.0.1/bin/cinder-rootwrap: Executable not found: qemu-img (filter match = qemu-img)\n'
2016-05-10 14:38:33.865 10357 ERROR oslo_messaging.rpc.dispatcher

A workaround is to manually install the qemu-utils package in the cinder-volume container. After performing the workaround I am able to successfully create cinder volumes backed by ceph, based on image source:

2016-05-10 14:49:28.317 10357 INFO cinder.volume.flows.manager.create_volume [req-fbc6fb79-cbc2-4fc7-97f8-d7b886f5b66c b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Volume af72d5a3-de07-4b7c-927c-f62ff2b18b97: being created as image with specification: {'status': u'creating', 'image_location': (u'rbd://1a99be1b-fd46-4b67-b2f1-3ea3efdd000f/glance/28bf4043-d449-416a-a994-234aa613ed7c/snap', None), 'volume_size': 20, 'volume_name': 'volume-af72d5a3-de07-4b7c-927c-f62ff2b18b97', 'image_id': u'28bf4043-d449-416a-a994-234aa613ed7c', 'image_service': <cinder.image.glance.GlanceImageService object at 0x7f648f70da90>, 'image_meta': {u'status': u'active', u'virtual_size': None, u'name': u'trusty64', u'tags': [], u'container_format': u'bare', u'created_at': datetime.datetime(2016, 5, 5, 21, 47, 13, tzinfo=<iso8601.Utc>), u'disk_format': u'qcow2', u'updated_at': datetime.datetime(2016, 5, 5, 21, 47, 20, tzinfo=<iso8601.Utc>), u'visibility': u'private', 'properties': {}, u'owner': u'a0adde92c1b74a67ad8c640168c28632', u'protected': False, u'id': u'28bf4043-d449-416a-a994-234aa613ed7c', u'file': u'/v2/images/28bf4043-d449-416a-a994-234aa613ed7c/file', u'checksum': u'e05b2f067604a1b4d0e9aa430b4e8721', u'min_disk': 0, u'direct_url': u'rbd://1a99be1b-fd46-4b67-b2f1-3ea3efdd000f/glance/28bf4043-d449-416a-a994-234aa613ed7c/snap', u'min_ram': 0, u'size': 259457536}}
2016-05-10 14:49:30.851 10357 INFO cinder.image.image_utils [req-fbc6fb79-cbc2-4fc7-97f8-d7b886f5b66c b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Image download 247.00 MB at 98.78 MB/s
2016-05-10 14:49:38.552 10357 INFO cinder.image.image_utils [req-fbc6fb79-cbc2-4fc7-97f8-d7b886f5b66c b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Converted 2252.00 MB image at 335.28 MB/s
2016-05-10 14:49:48.518 10357 INFO cinder.volume.flows.manager.create_volume [req-fbc6fb79-cbc2-4fc7-97f8-d7b886f5b66c b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Volume volume-af72d5a3-de07-4b7c-927c-f62ff2b18b97 (af72d5a3-de07-4b7c-927c-f62ff2b18b97): created successfully
2016-05-10 14:49:48.524 10357 INFO cinder.volume.manager [req-fbc6fb79-cbc2-4fc7-97f8-d7b886f5b66c b8e11a21c5f84dd2b629d480c00b9953 a0adde92c1b74a67ad8c640168c28632 - - -] Created volume successfully.

The right package is installed, but only if the cinder-volume service is also running on a host that's also used to provide an LVM backend.
https://github.com/openstack/openstack-ansible-os_cinder/blob/master/tasks/cinder_install_apt.yml#L39-L50
https://github.com/openstack/openstack-ansible-os_cinder/blob/master/vars/ubuntu-14.04.yml#L33

We'll need to ensure that the qemu-img package is installed wherever the cinder-volume service is installed, regardless of the back-end. Alternatively we need to implement back-end specific package lists.

This bug was introduced in the Mitaka development cycle by https://review.openstack.org/293836 so this issue is pertinent for Mitaka onwards.

Changed in openstack-ansible:
status: New → Confirmed
milestone: none → newton-1
importance: Undecided → High

I would suggest changing the task name for https://github.com/openstack/openstack-ansible-os_cinder/blob/master/tasks/cinder_install_apt.yml#L39-L50 to be 'Install apt packages for LVM backend'. Then adding another task with a corresponding var to 'Install apt packages commonly used for cinder-volume backends'. The conditional should just be 'when: inventory_hostname in groups['cinder_volume']'.

That should cover the apt package needs for most back-ends. Any special cases can be implemented afterwards using back-end specific lists.

Changed in openstack-ansible:
assignee: nobody → Flávio Ramalho (flaviosr)
Changed in openstack-ansible:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/316519
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=956683b908fd0210289fb0110e50a3b7c95f3cef
Submitter: Jenkins
Branch: master

commit 956683b908fd0210289fb0110e50a3b7c95f3cef
Author: Flávio Ramalho <email address hidden>
Date: Sun May 15 17:25:49 2016 -0300

    Added task to install apt packages idependent of the cinder-volume backend

    Packages needed by cinder-volume, idependent of the backend, were installed
    only when using a LVM backend. This change adds a new task and variable for
    installing/declaring packages commonly used between different cinder-volume
    backends.

    Change-Id: Idd7fd6f8be710c106210ec5bdc3ecda1071fc913
    Closes-Bug: #1580222

Changed in openstack-ansible:
status: In Progress → Fix Released
Greg Faust (gregfaust) wrote :

Works for me, thanks Flávio!

Flávio Ramalho (flaviosr) wrote :

Great Greg :)

Thanks for testing it!

Reviewed: https://review.openstack.org/317535
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_cinder/commit/?id=12a1e1c1185b46ecc7a7982d77b48f1b60fed4cc
Submitter: Jenkins
Branch: stable/mitaka

commit 12a1e1c1185b46ecc7a7982d77b48f1b60fed4cc
Author: Flávio Ramalho <email address hidden>
Date: Sun May 15 17:25:49 2016 -0300

    Added task to install apt packages idependent of the cinder-volume backend

    Packages needed by cinder-volume, idependent of the backend, were installed
    only when using a LVM backend. This change adds a new task and variable for
    installing/declaring packages commonly used between different cinder-volume
    backends.

    Change-Id: Idd7fd6f8be710c106210ec5bdc3ecda1071fc913
    Closes-Bug: #1580222
    (cherry picked from commit 956683b908fd0210289fb0110e50a3b7c95f3cef)

tags: added: in-stable-mitaka

This issue was fixed in the openstack/openstack-ansible-os_cinder 13.1.2 release.

This issue was fixed in the openstack/openstack-ansible-os_cinder 14.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers