when we hit this problem, I also observed RPC message timeout from the logfile.
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage [req-ed246967-33af-4f36-950b-a3fb935dee99 - - - - -] Error attempting to run shared_targets_online_data_migration: MessagingTimeout: Timed out waiting for a reply to message ID fcd7e35f688c497691a4f7f1333ee732 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage Traceback (most recent call last): 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/cinder/cmd/manage.py", line 351, in _run_migration 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage found, done = migration_meth(ctxt, count) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/cinder/cmd/manage.py", line 128, in shared_targets_online_data_migration 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage non_shared_hosts, total_vols_to_update = _get_non_shared_target_hosts(ctxt) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/cinder/cmd/manage.py", line 111, in _get_non_shared_target_hosts 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage capabilities = rpcapi.get_capabilities(ctxt, service.host, True) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/cinder/volume/rpcapi.py", line 341, in get_capabilities 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage return cctxt.call(ctxt, 'get_capabilities', discover=discover) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 179, in call 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage retry=self.retry) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/transport.py", line 133, in _send 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage retry=retry) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage call_monitor_timeout, retry=retry) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage call_monitor_timeout) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in wait 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage message = self.waiters.get(msg_id, timeout=timeout) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/venvs/cinder-18.1.8/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 336, in get 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage 'to message ID %s' % msg_id) 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage MessagingTimeout: Timed out waiting for a reply to message ID fcd7e35f688c497691a4f7f1333ee732 2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage
And I checked cinder message queue, it was only 1 message there.
So I can't explain why it timed out.
when we hit this problem, I also observed RPC message timeout from the logfile.
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage [req-ed246967- 33af-4f36- 950b-a3fb935dee 99 - - - - -] Error attempting to run shared_ targets_ online_ data_migration: MessagingTimeout: Timed out waiting for a reply to message ID fcd7e35f688c497 691a4f7f1333ee7 32 venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ cinder/ cmd/manage. py", line 351, in _run_migration meth(ctxt, count) venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ cinder/ cmd/manage. py", line 128, in shared_ targets_ online_ data_migration to_update = _get_non_ shared_ target_ hosts(ctxt) venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ cinder/ cmd/manage. py", line 111, in _get_non_ shared_ target_ hosts get_capabilitie s(ctxt, service.host, True) venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ cinder/ volume/ rpcapi. py", line 341, in get_capabilities venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ oslo_messaging/ rpc/client. py", line 179, in call venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ oslo_messaging/ transport. py", line 133, in _send venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ oslo_messaging/ _drivers/ amqpdriver. py", line 584, in send timeout, retry=retry) venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ oslo_messaging/ _drivers/ amqpdriver. py", line 573, in _send timeout) venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ oslo_messaging/ _drivers/ amqpdriver. py", line 459, in wait get(msg_ id, timeout=timeout) venvs/cinder- 18.1.8/ lib/python2. 7/site- packages/ oslo_messaging/ _drivers/ amqpdriver. py", line 336, in get 691a4f7f1333ee7 32
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage Traceback (most recent call last):
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage found, done = migration_
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage non_shared_hosts, total_vols_
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage capabilities = rpcapi.
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage return cctxt.call(ctxt, 'get_capabilities', discover=discover)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage retry=self.retry)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage retry=retry)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage call_monitor_
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage call_monitor_
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage message = self.waiters.
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage File "/openstack/
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage 'to message ID %s' % msg_id)
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage MessagingTimeout: Timed out waiting for a reply to message ID fcd7e35f688c497
2019-07-25 13:19:20.607 24665 ERROR cinder.cmd.manage
And I checked cinder message queue, it was only 1 message there.
So I can't explain why it timed out.