Hi, I can reproduce this bug:
1. kill the cinder-volume process
2. create a volume using "cinder create 1" command
3. sleep abount 10 seconds, start the cinder-volume process
The cinder-volume error log is:
ERROR cinder.volume.flows.manager.create_volume [req-c5d52847-c265-4247-b94d-eec17b3f3b91 0526988ada81462d9d522b4745a8f097 7b53d02d07414469804cb6fdd5a9a774] Volume 2280b2e3-1aa6-4f1c-a02b-f945346d5c6b: create failed
2014-03-12 16:49:51.235 DEBUG taskflow.storage [-] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' from (pid=8867) _with_connection /usr/local/lib/python2.7/dist-packages/taskflow/storage.py:88
2014-03-12 16:49:51.235 DEBUG taskflow.storage [-] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_task_detail' from (pid=8867) _with_connection /usr/local/lib/python2.7/dist-packages/taskflow/storage.py:88
2014-03-12 16:49:51.236 DEBUG taskflow.storage [-] No backend provided, not calling functor 'taskflow.storage.SingleThreadedStorage._save_flow_detail' from (pid=8867) _with_connection /usr/local/lib/python2.7/dist-packages/taskflow/storage.py:88
2014-03-12 16:49:51.236 ERROR cinder.openstack.common.rpc.amqp [req-c5d52847-c265-4247-b94d-eec17b3f3b91 0526988ada81462d9d522b4745a8f097 7b53d02d07414469804cb6fdd5a9a774] Exception during message handling
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/openstack/common/rpc/amqp.py", line 462, in _process_data
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp **args)
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 360, in create_volume
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp _run_flow()
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 353, in _run_flow
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp flow_engine.run()
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/taskflow/utils/lock_utils.py", line 53, in wrapper
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 111, in run
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp self._run()
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 121, in _run
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp self._revert(misc.Failure())
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/engine.py", line 78, in _revert
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp misc.Failure.reraise_if_any(failures.values())
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 558, in reraise_if_any
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp failures[0].reraise()
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/taskflow/utils/misc.py", line 565, in reraise
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp six.reraise(*self._exc_info)
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/taskflow/engines/action_engine/executor.py", line 36, in _execute_task
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp result = task.execute(**arguments)
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 582, in execute
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp raise exception.DriverNotInitialized()
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp DriverNotInitialized: Volume driver not ready.
2014-03-12 16:49:51.236 TRACE cinder.openstack.common.rpc.amqp
2014-03-12 16:49:51.371 DEBUG cinder.manager [req-6caa3e38-a86b-4ed8-bd44-6f81502e1cc4 None None] Notifying Schedulers of capabilities ... from (pid=8867) _publish_service_capabilities /opt/stack/cinder/cinder/manager.py:133
Hi, I can reproduce this bug:
1. kill the cinder-volume process
2. create a volume using "cinder create 1" command
3. sleep abount 10 seconds, start the cinder-volume process
The cinder-volume error log is: volume. flows.manager. create_ volume [req-c5d52847- c265-4247- b94d-eec17b3f3b 91 0526988ada81462 d9d522b4745a8f0 97 7b53d02d0741446 9804cb6fdd5a9a7 74] Volume 2280b2e3- 1aa6-4f1c- a02b-f945346d5c 6b: create failed storage. SingleThreadedS torage. _save_task_ detail' from (pid=8867) _with_connection /usr/local/ lib/python2. 7/dist- packages/ taskflow/ storage. py:88 storage. SingleThreadedS torage. _save_task_ detail' from (pid=8867) _with_connection /usr/local/ lib/python2. 7/dist- packages/ taskflow/ storage. py:88 storage. SingleThreadedS torage. _save_flow_ detail' from (pid=8867) _with_connection /usr/local/ lib/python2. 7/dist- packages/ taskflow/ storage. py:88 openstack. common. rpc.amqp [req-c5d52847- c265-4247- b94d-eec17b3f3b 91 0526988ada81462 d9d522b4745a8f0 97 7b53d02d0741446 9804cb6fdd5a9a7 74] Exception during message handling openstack. common. rpc.amqp Traceback (most recent call last): openstack. common. rpc.amqp File "/opt/stack/ cinder/ cinder/ openstack/ common/ rpc/amqp. py", line 462, in _process_data openstack. common. rpc.amqp **args) openstack. common. rpc.amqp File "/opt/stack/ cinder/ cinder/ openstack/ common/ rpc/dispatcher. py", line 172, in dispatch openstack. common. rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) openstack. common. rpc.amqp File "/opt/stack/ cinder/ cinder/ volume/ manager. py", line 360, in create_volume openstack. common. rpc.amqp _run_flow() openstack. common. rpc.amqp File "/opt/stack/ cinder/ cinder/ volume/ manager. py", line 353, in _run_flow openstack. common. rpc.amqp flow_engine.run() openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ utils/lock_ utils.py" , line 53, in wrapper openstack. common. rpc.amqp return f(*args, **kwargs) openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ engines/ action_ engine/ engine. py", line 111, in run openstack. common. rpc.amqp self._run() openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ engines/ action_ engine/ engine. py", line 121, in _run openstack. common. rpc.amqp self._revert( misc.Failure( )) openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ engines/ action_ engine/ engine. py", line 78, in _revert openstack. common. rpc.amqp misc.Failure. reraise_ if_any( failures. values( )) openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ utils/misc. py", line 558, in reraise_if_any openstack. common. rpc.amqp failures[ 0].reraise( ) openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ utils/misc. py", line 565, in reraise openstack. common. rpc.amqp six.reraise( *self._ exc_info) openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ engines/ action_ engine/ executor. py", line 36, in _execute_task openstack. common. rpc.amqp result = task.execute( **arguments) openstack. common. rpc.amqp File "/opt/stack/ cinder/ cinder/ volume/ flows/manager/ create_ volume. py", line 582, in execute openstack. common. rpc.amqp raise exception. DriverNotInitia lized() openstack. common. rpc.amqp DriverNotInitia lized: Volume driver not ready. openstack. common. rpc.amqp a86b-4ed8- bd44-6f81502e1c c4 None None] Notifying Schedulers of capabilities ... from (pid=8867) _publish_ service_ capabilities /opt/stack/ cinder/ cinder/ manager. py:133
ERROR cinder.
2014-03-12 16:49:51.235 DEBUG taskflow.storage [-] No backend provided, not calling functor 'taskflow.
2014-03-12 16:49:51.235 DEBUG taskflow.storage [-] No backend provided, not calling functor 'taskflow.
2014-03-12 16:49:51.236 DEBUG taskflow.storage [-] No backend provided, not calling functor 'taskflow.
2014-03-12 16:49:51.236 ERROR cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.371 DEBUG cinder.manager [req-6caa3e38-
And the volume status is error.