Share instance fails to create due to volume creation fails due to NoValidHost

[Matt Riedemann]

http://logs.openstack.org/23/260223/2/gate/gate-manila-tempest-dsvm-neutron/c507717/logs/screen-m-shr.txt.gz#_2015-12-22_17_12_41_096

2015-12-22 17:12:41.077 20339 ERROR manila.share.manager [req-ceb0f2db-4e24-4673-b826-b1236b7ee67b - - - - -] Share instance 49f47ffb-2a74-470d-8e86-d1f28786bdb4 failed on creation.
2015-12-22 17:12:41.078 20339 WARNING manila.share.manager [req-ceb0f2db-4e24-4673-b826-b1236b7ee67b - - - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher [req-ceb0f2db-4e24-4673-b826-b1236b7ee67b - - - - -] Exception during message handling: Failed to create volume
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher executor_callback))
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher executor_callback)
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 113, in wrapped
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher return f(self, *args, **kwargs)
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/utils.py", line 614, in wrapper
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher return func(self, *args, **kwargs)
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 800, in create_share_instance
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher {'status': constants.STATUS_ERROR}
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/manager.py", line 771, in create_share_instance
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher context, share_instance, share_server=share_server)
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/generic.py", line 122, in wrap
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kwargs)
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/generic.py", line 233, in create_share
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher volume = self._allocate_container(self.admin_context, share)
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/generic.py", line 526, in _allocate_container
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher msg_error=msg_error, msg_timeout=msg_timeout
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/manila/manila/share/drivers/generic.py", line 547, in _wait_for_available_volume
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher raise exception.ManilaException(msg_error)
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher ManilaException: Failed to create volume
2015-12-22 17:12:41.096 20339 ERROR oslo_messaging.rpc.dispatcher

The test that failed:

manila_tempest_tests.tests.api.admin.test_consistency_group_actions.ConsistencyGroupActionsTest.test_create_cg_from_multi_typed_populated_cgsnapshot_v2_4

http://logs.openstack.org/23/260223/2/gate/gate-manila-tempest-dsvm-neutron/c507717/console.html#_2015-12-22_17_27_08_814

This shows up 202 times in 7 days in voting jobs in the gate:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22create_share_instance%5C%22%20AND%20message:%5C%22_wait_for_available_volume%5C%22%20AND%20message:%5C%22ManilaException:%20Failed%20to%20create%20volume%5C%22%20AND%20tags:%5C%22screen-m-shr.txt%5C%22%20AND%20voting:%5C%221%5C%22

[Matt Riedemann (mriedem)]

Looks like the volume creation fails in cinder because there are no hosts available, so it's probably over quota:

http://logs.openstack.org/23/260223/2/gate/gate-manila-tempest-dsvm-neutron/c507717/logs/screen-c-sch.txt.gz?level=TRACE#_2015-12-22_17_12_33_877

[Hang Geng (gh965852614)]

Looks like the volume creation fails is cinder problem。
1.Connectivity check manila-share node and cinder between
2.check your cinder node create to volume or cinder-volume service is active/down？

[Ben Swartzlander (bswartz)]

I wouldn't surprise me if we're overloading cinder in some cases. I thought we took care of upping the space limit and the quota during Liberty but perhaps it needs to be upped again due to increased concurrency or new tests.

[Matt Riedemann (mriedem)]

This doesn't appear to be showing up in the gate anymore so we can close it.