can't handle rpc message when cinder-volume start
Bug #1271568 reported by
lirenke
This bug affects 2 people
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Cinder |
Fix Released
|
High
|
wanghong | ||
OpenStack Shared File Systems Service (Manila) |
Fix Released
|
High
|
Maurice Escher |
Bug Description
cinder-scheduler send a rpc message to cinder-volume which was killed just now.
Then when cinder-volume restart, it may receive the rpc message and begin to handle it before driver initialized.
Most functions in manager need to judge wether the driver has been initialized, so the rpc request would be reject. But those requests shoud be handle in fact.
The reason is, in service.py, we create consumer before manager call "init_host()", and we call "self.driver.
description: | updated |
Changed in cinder: | |
assignee: | nobody → lirenke (lvhancy) |
Changed in cinder: | |
status: | New → Confirmed |
Changed in cinder: | |
status: | Confirmed → Incomplete |
Changed in cinder: | |
milestone: | none → icehouse-rc1 |
importance: | Undecided → High |
Changed in cinder: | |
status: | Fix Committed → Fix Released |
Changed in cinder: | |
milestone: | icehouse-rc1 → 2014.1 |
Changed in manila: | |
importance: | Undecided → High |
assignee: | nobody → Maurice Escher (maurice-escher) |
To post a comment you must log in.
Hi, I can reproduce this bug:
1. kill the cinder-volume process
2. create a volume using "cinder create 1" command
3. sleep abount 10 seconds, start the cinder-volume process
The cinder-volume error log is: volume. flows.manager. create_ volume [req-c5d52847- c265-4247- b94d-eec17b3f3b 91 0526988ada81462 d9d522b4745a8f0 97 7b53d02d0741446 9804cb6fdd5a9a7 74] Volume 2280b2e3- 1aa6-4f1c- a02b-f945346d5c 6b: create failed storage. SingleThreadedS torage. _save_task_ detail' from (pid=8867) _with_connection /usr/local/ lib/python2. 7/dist- packages/ taskflow/ storage. py:88 storage. SingleThreadedS torage. _save_task_ detail' from (pid=8867) _with_connection /usr/local/ lib/python2. 7/dist- packages/ taskflow/ storage. py:88 storage. SingleThreadedS torage. _save_flow_ detail' from (pid=8867) _with_connection /usr/local/ lib/python2. 7/dist- packages/ taskflow/ storage. py:88 openstack. common. rpc.amqp [req-c5d52847- c265-4247- b94d-eec17b3f3b 91 0526988ada81462 d9d522b4745a8f0 97 7b53d02d0741446 9804cb6fdd5a9a7 74] Exception during message handling openstack. common. rpc.amqp Traceback (most recent call last): openstack. common. rpc.amqp File "/opt/stack/ cinder/ cinder/ openstack/ common/ rpc/amqp. py", line 462, in _process_data openstack. common. rpc.amqp **args) openstack. common. rpc.amqp File "/opt/stack/ cinder/ cinder/ openstack/ common/ rpc/dispatcher. py", line 172, in dispatch openstack. common. rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs) openstack. common. rpc.amqp File "/opt/stack/ cinder/ cinder/ volume/ manager. py", line 360, in create_volume openstack. common. rpc.amqp _run_flow() openstack. common. rpc.amqp File "/opt/stack/ cinder/ cinder/ volume/ manager. py", line 353, in _run_flow openstack. common. rpc.amqp flow_engine.run() openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ utils/lock_ utils.py" , line 53, in wrapper openstack. common. rpc.amqp return f(*args, **kwargs) openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ engines/ action_ engine/ engine. py", line 111, in run openstack. common. rpc.amqp self._run() openstack. common. rpc.amqp File "/usr/local/ lib/python2. 7/dist- packages/ taskflow/ engines/ action_ engine/ engine. py", line 121, in _run openstack. common. rpc.amqp self._revert( misc.Failure( ))
ERROR cinder.
2014-03-12 16:49:51.235 DEBUG taskflow.storage [-] No backend provided, not calling functor 'taskflow.
2014-03-12 16:49:51.235 DEBUG taskflow.storage [-] No backend provided, not calling functor 'taskflow.
2014-03-12 16:49:51.236 DEBUG taskflow.storage [-] No backend provided, not calling functor 'taskflow.
2014-03-12 16:49:51.236 ERROR cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49:51.236 TRACE cinder.
2014-03-12 16:49...