2014-04-07 10:12:50.670 DEBUG cinder.volume.flows.common [req-4d18bab1-d028-471d-af74-0ce2a757797b c49326b373c5468093dbd3a8af79e73c 05d1678017714e3ca1c524924ca0ed27] Updating volume: 29e028b7-520b-46f3-963c-980e3bfceab9 with {'status': 'error'} due to: ??? from (pid=3682) error_out_volume /opt/stack/cinder/cinder/volume/flows/common.py:87 2014-04-07 10:12:50.719 ERROR cinder.volume.flows.manager.create_volume [req-4d18bab1-d028-471d-af74-0ce2a757797b c49326b373c5468093dbd3a8af79e73c 05d1678017714e3ca1c524924ca0ed27] Volume 29e028b7-520b-46f3-963c-980e3bfceab9: create failed 2014-04-07 10:12:50.720 DEBUG taskflow.storage [req-4d18bab1-d028-471d-af74-0ce2a757797b c49326b373c5468093dbd3a8af79e73c 05d1678017714e3ca1c524924ca0ed27] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' from (pid=3682) _with_connection /opt/stack/taskflow/taskflow/storage.py:86 2014-04-07 10:12:50.720 DEBUG taskflow.storage [req-4d18bab1-d028-471d-af74-0ce2a757797b c49326b373c5468093dbd3a8af79e73c 05d1678017714e3ca1c524924ca0ed27] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' from (pid=3682) _with_connection /opt/stack/taskflow/taskflow/storage.py:86 2014-04-07 10:12:50.721 DEBUG taskflow.storage [req-4d18bab1-d028-471d-af74-0ce2a757797b c49326b373c5468093dbd3a8af79e73c 05d1678017714e3ca1c524924ca0ed27] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_flow_detail' from (pid=3682) _with_connection /opt/stack/taskflow/taskflow/storage.py:86 2014-04-07 10:12:50.721 DEBUG cinder.openstack.common.lockutils [req-4d18bab1-d028-471d-af74-0ce2a757797b c49326b373c5468093dbd3a8af79e73c 05d1678017714e3ca1c524924ca0ed27] Released file lock "fa316dd6-d31c-41c1-baac-270eb0389be7-delete_volume" at /opt/stack/data/cinder/cinder-fa316dd6-d31c-41c1-baac-270eb0389be7-delete_volume for method "_run_flow_locked"... from (pid=3682) inner /opt/stack/cinder/cinder/openstack/common/lockutils.py:239 2014-04-07 10:12:50.722 ERROR oslo.messaging._executors.base [req-4d18bab1-d028-471d-af74-0ce2a757797b c49326b373c5468093dbd3a8af79e73c 05d1678017714e3ca1c524924ca0ed27] Exception during message handling 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base Traceback (most recent call last): 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_executors/base.py", line 36, in _dispatch 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base incoming.reply(self.callback(incoming.ctxt, incoming.message)) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in __call__ 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base return self._dispatch(endpoint, method, ctxt, args) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base result = getattr(endpoint, method)(ctxt, **new_args) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/manager.py", line 365, in create_volume 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base _run_flow_locked() 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/openstack/common/lockutils.py", line 233, in inner 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base retval = f(*args, **kwargs) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/manager.py", line 360, in _run_flow_locked 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base _run_flow() 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/manager.py", line 356, in _run_flow 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base flow_engine.run() 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/taskflow/taskflow/utils/lock_utils.py", line 54, in wrapper 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base return f(*args, **kwargs) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/taskflow/taskflow/engines/action_engine/engine.py", line 94, in run 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base self._run() 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/taskflow/taskflow/engines/action_engine/engine.py", line 109, in _run 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base misc.Failure.reraise_if_any(failures.values()) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/taskflow/taskflow/utils/misc.py", line 586, in reraise_if_any 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base failures[0].reraise() 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/taskflow/taskflow/utils/misc.py", line 593, in reraise 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base six.reraise(*self._exc_info) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/taskflow/taskflow/engines/action_engine/executor.py", line 34, in _execute_task 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base result = task.execute(**arguments) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 594, in execute 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base **volume_spec) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 433, in _create_from_source_volume 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py", line 1360, in create_cloned_volume 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base self._create_cloned_volume(volume, src_vref) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py", line 1332, in _create_cloned_volume 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base self._verify_volume_creation(volume) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py", line 246, in _verify_volume_creation 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base self._select_ds_for_volume(volume) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py", line 468, in _select_ds_for_volume 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base retrv_result = self.volumeops.get_hosts() 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/drivers/vmware/volumeops.py", line 121, in get_hosts 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base 'HostSystem', self._max_objects) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/drivers/vmware/api.py", line 252, in invoke_api 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base return _invoke_api(module, method, *args, **kwargs) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/drivers/vmware/api.py", line 89, in func 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base return timer.wait() 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base return hubs.get_hub().switch() 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base return self.greenlet.switch() 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/openstack/common/loopingcall.py", line 123, in _inner 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base idle = self.f(*self.args, **self.kw) 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base File "/opt/stack/cinder/cinder/volume/drivers/vmware/api.py", line 82, in _func 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base raise excep 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base VimFaultException: Server raised fault: 'The session is not authenticated.' 2014-04-07 10:12:50.722 TRACE oslo.messaging._executors.base 2014-04-07 10:12:50.724 ERROR oslo.messaging._drivers.common [req-4d18bab1-d028-471d-af74-0ce2a757797b c49326b373c5468093dbd3a8af79e73c 05d1678017714e3ca1c524924ca0ed27] Returning exception Server raised fault: 'The session is not authenticated.' to caller 2014-04-07 10:12:50.724 ERROR oslo.messaging._drivers.common [req-4d18bab1-d028-471d-af74-0ce2a757797b c49326b373c5468093dbd3a8af79e73c 05d1678017714e3ca1c524924ca0ed27] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_executors/base.py", line 36, in _dispatch\n incoming.reply(self.callback(incoming.ctxt, incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in __call__\n return self._dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 365, in create_volume\n _run_flow_locked()\n', ' File "/opt/stack/cinder/cinder/openstack/common/lockutils.py", line 233, in inner\n retval = f(*args, **kwargs)\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 360, in _run_flow_locked\n _run_flow()\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 356, in _run_flow\n flow_engine.run()\n', ' File "/opt/stack/taskflow/taskflow/utils/lock_utils.py", line 54, in wrapper\n return f(*args, **kwargs)\n', ' File "/opt/stack/taskflow/taskflow/engines/action_engine/engine.py", line 94, in run\n self._run()\n', ' File "/opt/stack/taskflow/taskflow/engines/action_engine/engine.py", line 109, in _run\n misc.Failure.reraise_if_any(failures.values())\n', ' File "/opt/stack/taskflow/taskflow/utils/misc.py", line 586, in reraise_if_any\n failures[0].reraise()\n', ' File "/opt/stack/taskflow/taskflow/utils/misc.py", line 593, in reraise\n six.reraise(*self._exc_info)\n', ' File "/opt/stack/taskflow/taskflow/engines/action_engine/executor.py", line 34, in _execute_task\n result = task.execute(**arguments)\n', ' File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 594, in execute\n **volume_spec)\n', ' File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 433, in _create_from_source_volume\n model_update = self.driver.create_cloned_volume(volume_ref, srcvol_ref)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py", line 1360, in create_cloned_volume\n self._create_cloned_volume(volume, src_vref)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py", line 1332, in _create_cloned_volume\n self._verify_volume_creation(volume)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py", line 246, in _verify_volume_creation\n self._select_ds_for_volume(volume)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/vmware/vmdk.py", line 468, in _select_ds_for_volume\n retrv_result = self.volumeops.get_hosts()\n', ' File "/opt/stack/cinder/cinder/volume/drivers/vmware/volumeops.py", line 121, in get_hosts\n \'HostSystem\', self._max_objects)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/vmware/api.py", line 252, in invoke_api\n return _invoke_api(module, method, *args, **kwargs)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/vmware/api.py", line 89, in func\n return timer.wait()\n', ' File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait\n return hubs.get_hub().switch()\n', ' File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch\n return self.greenlet.switch()\n', ' File "/opt/stack/cinder/cinder/openstack/common/loopingcall.py", line 123, in _inner\n idle = self.f(*self.args, **self.kw)\n', ' File "/opt/stack/cinder/cinder/volume/drivers/vmware/api.py", line 82, in _func\n raise excep\n', "VimFaultException: Server raised fault: 'The session is not authenticated.'\n"] Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_executors/base.py", line 46, in _dispatch incoming.reply(failure=exc_info) File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 63, in reply self._send_reply(conn, reply, failure, log_failure=log_failure) File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 44, in _send_reply log_failure) File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 327, in serialize_remote_exception json_data = jsonutils.dumps(data) File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps return json.dumps(value, default=default, **kwargs) File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps **kw).encode(obj) File "/usr/lib/python2.7/json/encoder.py", line 201, in encode chunks = self.iterencode(o, _one_shot=True) File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode return _iterencode(o, 0) ValueError: Circular reference detected 2014-04-07 10:12:57.813 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._publish_service_capabilities from (pid=3682) run_periodic_tasks /opt/stack/cinder/cinder/openstack/common/periodic_task.py:178 2014-04-07 10:12:57.813 DEBUG cinder.manager [-] Notifying Schedulers of capabilities ... from (pid=3682) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:128 2014-04-07 10:12:57.814 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is e78f250e552b4095812a1b94fdcad4ae. from (pid=3682) _add_unique_id /usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqp.py:333 2014-04-07 10:12:57.816 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._report_driver_status from (pid=3682) run_periodic_tasks /opt/stack/cinder/cinder/openstack/common/periodic_task.py:178 2014-04-07 10:13:57.816 DEBUG cinder.openstack.common.periodic_task [-] Running periodic task VolumeManager._publish_service_capabilities from (pid=3682) run_periodic_tasks /opt/stack/cinder/cinder/openstack/common/periodic_task.py:178) n-crt 9$(L) n-net 10$(L) n-sch 11-$(L) c-api 12$(L) c-sch 13$(L) c-vol* 2014-04-07 10:13:57.817 DEBUG cinder.manager [-] Notifying Schedulers of capabilities ... from (pid=3682) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:128