Volumes failed to create, stay in Creating status

[Tatyanka]

{"build_id": "2014-05-05_00-15-43", "mirantis": "yes", "build_number": "180", "ostf_sha": "134765fcb5a07dce0cd1bb399b2290c988c3c63b", "nailgun_sha": "2de1dcf9fa3fc1521999bff6377eaa6f01d825aa", "production": "docker", "api": "1.0", "fuelmain_sha": "95c35c199c2efc03fb105d090c5a42525430b7b3", "astute_sha": "3cffebde1e5452f5dbf8f744c6525fc36c7afbf3", "release": "5.0", "fuellib_sha": "2348fae80b21c3ec9e5f520395eea2943a510f3d"}

Steps to reproduce:
Deploy simple env on neutron gre with sahara, murano, ceilometer
add 1 controller 3 computes and 1 node with cinder/mongo role

1.Deploy environment
2. Run OSTf
3. Try to create volume in horizon

Actual result:
Volumes do not get available status and stay in creating

[Mike Scherbakov (mihgen)]

I had similar env with HA (only 1 controller), compute & cinder roles on one node, and faced same issue. I was not able to find exceptions or other errors in logs. Only following warnings:
2014-05-05T20:32:23.699839+00:00 warning: AUDIT: vol={'migration_status': None, 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2014, 5, 5, 20, 10, 22), 'provider_geometry': None, 'snapshot_id': None, 'ec2_id': None, 'mountpoint': None, 'deleted_at': None, 'id': u'e1964779-aa46-4d74-8af9-3006cea74e70', 'size': 1L, 'user_id': u'e17f3adee0c242518160d5e6507d68d0', 'attach_time': None, 'attached_host': None, 'display_description': None, 'volume_admin_metadata': [], 'encryption_key_id': None, 'project_id': u'66f1e01a30214b1e9f9ca86bc1bd201e', 'launched_at': None, 'scheduled_at': datetime.datetime(2014, 5, 5, 20, 10, 22), 'status': u'creating', 'volume_type_id': None, 'deleted': False, 'provider_location': None, 'host': u'node-2.domain.tld', 'source_volid': None, 'provider_auth': None, 'display_name': u'ost1_test-volume559040366', 'instance_uuid': None, 'bootable': False, 'created_at': datetime.datetime(2014, 5, 5, 20, 10, 21), 'attach_status': u'detached', 'volume_type': None, '_name_id': None, 'volume_metadata': []}

[Mike Scherbakov (mihgen)]

With enabled debug logs for scheduler:
2014-05-05 21:03:12.721 22657 DEBUG cinder.scheduler.filter_scheduler [req-2c999eef-ceea-4f6f-936e-f237eda72120 e17f3adee0c2425181
60d5e6507d68d0 66f1e01a30214b1e9f9ca86bc1bd201e - - -] Choosing node-2.domain.tld _choose_top_host /usr/lib/python2.6/site-package
s/cinder/scheduler/filter_scheduler.py:276
2014-05-05 21:03:12.863 22657 DEBUG taskflow.storage [req-2c999eef-ceea-4f6f-936e-f237eda72120 e17f3adee0c242518160d5e6507d68d0 66
f1e01a30214b1e9f9ca86bc1bd201e - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_
detail' _with_connection /usr/lib/python2.6/site-packages/taskflow/storage.py:88
2014-05-05 21:03:12.863 22657 DEBUG taskflow.storage [req-2c999eef-ceea-4f6f-936e-f237eda72120 e17f3adee0c242518160d5e6507d68d0 66
f1e01a30214b1e9f9ca86bc1bd201e - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_
detail' _with_connection /usr/lib/python2.6/site-packages/taskflow/storage.py:88
2014-05-05 21:03:12.864 22657 DEBUG taskflow.storage [req-2c999eef-ceea-4f6f-936e-f237eda72120 e17f3adee0c242518160d5e6507d68d0 66
f1e01a30214b1e9f9ca86bc1bd201e - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_flow_
detail' _with_connection /usr/lib/python2.6/site-packages/taskflow/storage.py:88
2014-05-05 21:03:39.887 22657 WARNING cinder.context [-] Arguments dropped when creating context: {'user': None, 'tenant': None, '
user_identity': u'- - - - -'}

cinder-volume has no activity, only reports about capacity.

[Vladimir Kuklin (vkuklin)]

for now debugging shows that there are no messages sent to cinder queues for particular cinder-volume host. e.g. stop the cinder-volume service on the cinder node, try to create a volume. it is scheduled. but there are no messages in the queues.

[Tatyanka (tatyana-leontovich)]

btw our system tests shows that this can be happens on different stage of volume creation and lead to the error status
2014-05-06T12:04:35.284005+00:00 debug: 2014-05-06 12:04:33.427 17859 WARNING cinder.context [-] Arguments dropped when creating context: {'user': None, 'tenant': None, 'user_identity': u'- - - - -'}
2014-05-06T11:59:45.186512+00:00 debug: 2014-05-06 11:59:43.344 17859 DEBUG taskflow.storage [req-9cea850a-857a-439f-b149-1749eae7a756 007bb41bb2a148e3bf7046ffc08f1a07 527c6a4ceca846a587d95ba4694f3608 - - -] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_flow_detail' _with_connection /usr/lib/python2.6/site-packages/taskflow/storage.py:88
2014-05-06T12:00:25.204025+00:00 debug: 2014-05-06 12:00:15.284 17859 WARNING cinder.context [-] Arguments dropped when creating context: {'user': None, 'tenant': None, 'user_identity': u'- - - - -'}
2014-05-06T12:00:25.204025+00:00 debug: 2014-05-06 12:00:15.285 17859 DEBUG cinder.scheduler.host_manager [req-48d32b05-ce67-40ec-846a-4a990c62eccd - - - - -] Received volume service update from node-1.test.domain.local. update_service_capabilities /usr/lib/python2.6/site-packages/cinder/scheduler/host_manager.py:273
2014-05-06T12:01:25.212577+00:00 debug: 2014-05-06 12:01:15.290 17859 WARNING cinder.context [-] Arguments dropped when creating context: {'user': None, 'tenant': None, 'user_identity': u'- - - - -'}

http://jenkins-product.srt.mirantis.net:8080/view/0_0_swarm/job/master_fuelmain.system_test.centos.thread_2/39/

[Vladimir Kuklin (vkuklin)]

this warning does not like the problem. it is harmless artifact - there are plenty of issues regarding this artifact in many openstack projects. this happens when you create volume using empty (admin ?) tenant, I guess.

[Vladimir Kuklin (vkuklin)]

interesting moment:

I created a 10T sparse file and used it as PV for cinder VG on the controller node, which did not have cinder-volume role. it creates volumes just fine. may be this is some kind of networking issue

[Andrew Woodward (xarses)]

cinder-volume with Ceph backend does work

[Vladimir Kuklin (vkuklin)]

the same is true for the compute node - manual creation of cinder VG and configuration of cinder-volume is ok

[Vladimir Kuklin (vkuklin)]

JFYI - I used have following setup:

compute+controller+(cinder+mongo)

[Vladimir Kuklin (vkuklin)]

it is control_exchange variable in the config. fix on his way.

[Andrew Woodward (xarses)]

https://review.openstack.org/#/c/92484/

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/92484
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=76d7bf2d268150d738d07b0b4501f3e5d404deae
Submitter: Jenkins
Branch: master

commit 76d7bf2d268150d738d07b0b4501f3e5d404deae
Author: Vladimir Kuklin <email address hidden>
Date: Wed May 7 02:41:03 2014 +0400

    Set identical control_exchange for all cinder nodes

    Otherwise, nodes will look into different queues
    and this will lead to volume creation failure.

    Change-Id: Ia0c97700543d440e03746bbadc4a90d25121fa90
    Closes-Bug: #1316204

[Andrey Sledzinskiy (asledzinskiy)]

verified on {"build_id": "2014-05-08_01-10-31", "mirantis": "yes", "build_number": "188", "ostf_sha": "fe718434f88f2ab167779770828a195f06eb29f8", "nailgun_sha": "82b9d42a7a5e9aa1caf6b2779c45ca045cad0ad2", "production": "docker", "api": "1.0", "fuelmain_sha": "97d7f6d5461db3afc27f58160cf9f6985230d255", "astute_sha": "9c83d3ecec69df03cd94620e2df92249ba4ec786", "release": "5.0", "fuellib_sha": "fd31d9a8f85136347b60377df00df6728eda14ca"}