| 2015-09-08 19:05:56 |
Sean Severson |
bug |
|
|
added bug |
| 2015-09-08 22:37:53 |
Sean Severson |
description |
If using an environment where there are both nodes that share the nova instances directory and nodes that do NOT share the directory, any attempt to block-migrate to non-shared nodes from the controller node will fail.
Example:
1. Create an environment with one controller node and two cpu nodes.
2. Have the controller node act as an NFS server by adding the instances directory to /etc/exports/
3. Add this new share to the fstab of just one of the CPU nodes.
4. Mount the new share after stacking the appropriate CPU node.
5. Stack the unshared CPU node.
The next step applied to my scenario, but may not be necessary.
6. Create a bootable volume in Cinder.
7. Launch an instance from that volume, using a flavor that does NOT create a local disk. (This landed on the controller node in my scenario)
8. Once the instance is running, attempt to block-migrate to the unshared node. (nova live-migration --block-migrate <instance> <host>)
In the past it was possible to block-migrate to and from the unshared node, then migrate without block between the shared nodes. Now in Liberty (master) the following errors appears:
2015-09-08 12:09:17.052 ERROR oslo_messaging.rpc.dispatcher [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Exception during message handling: cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
executor_callback))
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
executor_callback)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
result = func(ctxt, **new_args)
File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
payload)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
return f(self, context, *args, **kw)
File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
raise exception.InvalidLocalStorage(reason=reason, path=source)
InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data=dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=self.retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher timeout=timeout, retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise result
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise exception.InvalidLocalStorage(reason=reason, path=source)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.055 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Returning exception cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
executor_callback))
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
executor_callback)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
result = func(ctxt, **new_args)
File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
payload)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
return f(self, context, *args, **kw)
File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
raise exception.InvalidLocalStorage(reason=reason, path=source)
InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
to caller
2015-09-08 12:09:17.056 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination\n dest_check_data)\n', ' File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source\n dest_check_data=dest_check_data)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call\n retry=self.retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send\n timeout=timeout, retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send\n retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send\n raise result\n', 'InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.\nTraceback (most recent call last):\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n\n File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source\n block_device_info)\n\n File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source\n raise exception.InvalidLocalStorage(reason=reason, path=source)\n\nInvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.\n\n']
This check appears to only check the source, and is not actually testing to see if the storage is shared between the two nodes. |
If using an environment where there are both nodes that share the nova instances directory and nodes that do NOT share the directory, any attempt to block-migrate to non-shared nodes from the controller node will fail.
Example:
1. Create an environment with one controller node and two cpu nodes.
2. Have the controller node act as an NFS server by adding the instances directory to /etc/exports/
3. Add this new share to the fstab of just one of the CPU nodes.
4. Mount the new share after stacking the appropriate CPU node.
5. Stack the unshared CPU node.
The next step applied to my scenario, but may not be necessary.
6. Create a bootable volume in Cinder.
7. Launch an instance from that volume, using a flavor that does NOT create a local disk. (This landed on the controller node in my scenario)
8. Once the instance is running, attempt to block-migrate to the unshared node. (nova live-migration --block-migrate <instance> <host>)
In the past it was possible to block-migrate to and from the unshared node, then migrate without block between the shared nodes. Now in Liberty (master) the following errors appears:
2015-09-08 12:09:17.052 ERROR oslo_messaging.rpc.dispatcher [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Exception during message handling: cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
executor_callback))
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
executor_callback)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
result = func(ctxt, **new_args)
File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
payload)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
return f(self, context, *args, **kw)
File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
raise exception.InvalidLocalStorage(reason=reason, path=source)
InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data=dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=self.retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher timeout=timeout, retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise result
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise exception.InvalidLocalStorage(reason=reason, path=source)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.055 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Returning exception cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
executor_callback))
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
executor_callback)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
result = func(ctxt, **new_args)
File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
payload)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
return f(self, context, *args, **kw)
File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
raise exception.InvalidLocalStorage(reason=reason, path=source)
InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
to caller
2015-09-08 12:09:17.056 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination\n dest_check_data)\n', ' File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source\n dest_check_data=dest_check_data)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call\n retry=self.retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send\n timeout=timeout, retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send\n retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send\n raise result\n', 'InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.\nTraceback (most recent call last):\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n\n File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source\n block_device_info)\n\n File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source\n raise exception.InvalidLocalStorage(reason=reason, path=source)\n\nInvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.\n\n']
This check appears to only check the source, and is not actually testing to see if the storage is shared between the two nodes. |
|
| 2015-09-08 22:43:19 |
Sean Severson |
description |
If using an environment where there are both nodes that share the nova instances directory and nodes that do NOT share the directory, any attempt to block-migrate to non-shared nodes from the controller node will fail.
Example:
1. Create an environment with one controller node and two cpu nodes.
2. Have the controller node act as an NFS server by adding the instances directory to /etc/exports/
3. Add this new share to the fstab of just one of the CPU nodes.
4. Mount the new share after stacking the appropriate CPU node.
5. Stack the unshared CPU node.
The next step applied to my scenario, but may not be necessary.
6. Create a bootable volume in Cinder.
7. Launch an instance from that volume, using a flavor that does NOT create a local disk. (This landed on the controller node in my scenario)
8. Once the instance is running, attempt to block-migrate to the unshared node. (nova live-migration --block-migrate <instance> <host>)
In the past it was possible to block-migrate to and from the unshared node, then migrate without block between the shared nodes. Now in Liberty (master) the following errors appears:
2015-09-08 12:09:17.052 ERROR oslo_messaging.rpc.dispatcher [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Exception during message handling: cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
executor_callback))
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
executor_callback)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
result = func(ctxt, **new_args)
File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
payload)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
return f(self, context, *args, **kw)
File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
raise exception.InvalidLocalStorage(reason=reason, path=source)
InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data=dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=self.retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher timeout=timeout, retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise result
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise exception.InvalidLocalStorage(reason=reason, path=source)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.055 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Returning exception cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
executor_callback))
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
executor_callback)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
result = func(ctxt, **new_args)
File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
payload)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
return f(self, context, *args, **kw)
File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
raise exception.InvalidLocalStorage(reason=reason, path=source)
InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
to caller
2015-09-08 12:09:17.056 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination\n dest_check_data)\n', ' File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source\n dest_check_data=dest_check_data)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call\n retry=self.retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send\n timeout=timeout, retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send\n retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send\n raise result\n', 'InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.\nTraceback (most recent call last):\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n\n File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source\n block_device_info)\n\n File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source\n raise exception.InvalidLocalStorage(reason=reason, path=source)\n\nInvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.\n\n']
This check appears to only check the source, and is not actually testing to see if the storage is shared between the two nodes. |
Nova is not allowing any block-migrate transactions to occur due to false positives when testing for shared storage. This is occurring with even a simple two node setup.
------Original Entry------
If using an environment where there are both nodes that share the nova instances directory and nodes that do NOT share the directory, any attempt to block-migrate to non-shared nodes from the controller node will fail.
Example:
1. Create an environment with one controller node and two cpu nodes.
2. Have the controller node act as an NFS server by adding the instances directory to /etc/exports/
3. Add this new share to the fstab of just one of the CPU nodes.
4. Mount the new share after stacking the appropriate CPU node.
5. Stack the unshared CPU node.
The next step applied to my scenario, but may not be necessary.
6. Create a bootable volume in Cinder.
7. Launch an instance from that volume, using a flavor that does NOT create a local disk. (This landed on the controller node in my scenario)
8. Once the instance is running, attempt to block-migrate to the unshared node. (nova live-migration --block-migrate <instance> <host>)
In the past it was possible to block-migrate to and from the unshared node, then migrate without block between the shared nodes. Now in Liberty (master) the following errors appears:
2015-09-08 12:09:17.052 ERROR oslo_messaging.rpc.dispatcher [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Exception during message handling: cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
executor_callback))
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
executor_callback)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
result = func(ctxt, **new_args)
File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
payload)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
return f(self, context, *args, **kw)
File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
raise exception.InvalidLocalStorage(reason=reason, path=source)
InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data=dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=self.retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher timeout=timeout, retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise result
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise exception.InvalidLocalStorage(reason=reason, path=source)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.055 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Returning exception cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
executor_callback))
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
executor_callback)
File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
result = func(ctxt, **new_args)
File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
payload)
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
return f(self, context, *args, **kw)
File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
kwargs['instance'], e, sys.exc_info())
File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
return function(self, context, *args, **kwargs)
File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
block_device_info)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
raise exception.InvalidLocalStorage(reason=reason, path=source)
InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
to caller
2015-09-08 12:09:17.056 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination\n dest_check_data)\n', ' File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source\n dest_check_data=dest_check_data)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call\n retry=self.retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send\n timeout=timeout, retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send\n retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send\n raise result\n', 'InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.\nTraceback (most recent call last):\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n\n File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source\n block_device_info)\n\n File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source\n raise exception.InvalidLocalStorage(reason=reason, path=source)\n\nInvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.\n\n']
This check appears to only check the source, and is not actually testing to see if the storage is shared between the two nodes. |
|
| 2015-09-10 16:34:19 |
Matt Riedemann |
tags |
|
migration volumes |
|
| 2015-09-10 17:34:50 |
Sean Severson |
attachment added |
|
Cinder config file https://bugs.launchpad.net/nova/+bug/1493512/+attachment/4460593/+files/cinder.conf |
|
| 2015-09-10 17:35:18 |
Sean Severson |
attachment added |
|
Nova config file https://bugs.launchpad.net/nova/+bug/1493512/+attachment/4460594/+files/nova.conf |
|
| 2015-09-17 18:46:25 |
Matt Riedemann |
tags |
migration volumes |
liberty-rc-potential migration volumes |
|
| 2015-09-18 14:05:40 |
Matt Riedemann |
tags |
liberty-rc-potential migration volumes |
liberty-rc-potential live-migration migration volumes |
|
| 2015-09-18 15:38:32 |
John Garbutt |
nova: importance |
Undecided |
High |
|
| 2015-09-18 15:38:52 |
John Garbutt |
nova: importance |
High |
Undecided |
|
| 2015-09-23 15:49:16 |
Matt Riedemann |
tags |
liberty-rc-potential live-migration migration volumes |
liberty-backport-potential live-migration migration volumes |
|
| 2015-09-30 11:29:34 |
Pawel Koniszewski |
nova: status |
New |
Incomplete |
|
| 2015-09-30 11:29:51 |
Pawel Koniszewski |
tags |
liberty-backport-potential live-migration migration volumes |
liberty-backport-potential live-migrate volumes |
|
| 2015-11-06 17:17:31 |
Paul Murray |
tags |
liberty-backport-potential live-migrate volumes |
liberty-backport-potential live-migration volumes |
|
| 2015-11-17 13:56:40 |
lvmxh |
nova: assignee |
|
lvmxh (shaohef) |
|
| 2015-12-03 09:21:14 |
lvmxh |
nova: assignee |
lvmxh (shaohef) |
|
|
| 2016-02-02 04:17:21 |
Launchpad Janitor |
nova: status |
Incomplete |
Expired |
|
