OpenStack Compute (nova)

n-cpu misreports shared storage when attempting to block migrate

Bug #1493512 reported by Sean Severson
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Expired
Undecided
Unassigned

Bug Description

Nova is not allowing any block-migrate transactions to occur due to false positives when testing for shared storage. This is occurring with even a simple two node setup.

------Original Entry------

If using an environment where there are both nodes that share the nova instances directory and nodes that do NOT share the directory, any attempt to block-migrate to non-shared nodes from the controller node will fail.

Example:
1. Create an environment with one controller node and two cpu nodes.
2. Have the controller node act as an NFS server by adding the instances directory to /etc/exports/
3. Add this new share to the fstab of just one of the CPU nodes.
4. Mount the new share after stacking the appropriate CPU node.
5. Stack the unshared CPU node.

The next step applied to my scenario, but may not be necessary.
6. Create a bootable volume in Cinder.
7. Launch an instance from that volume, using a flavor that does NOT create a local disk. (This landed on the controller node in my scenario)
8. Once the instance is running, attempt to block-migrate to the unshared node. (nova live-migration --block-migrate <instance> <host>)

In the past it was possible to block-migrate to and from the unshared node, then migrate without block between the shared nodes. Now in Liberty (master) the following errors appears:

2015-09-08 12:09:17.052 ERROR oslo_messaging.rpc.dispatcher [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Exception during message handling: cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
    executor_callback))

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
    executor_callback)

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
    payload)

  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
    return f(self, context, *args, **kw)

  File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
    kwargs['instance'], e, sys.exc_info())

  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
    block_device_info)

  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
    raise exception.InvalidLocalStorage(reason=reason, path=source)

InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.

2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher dest_check_data=dest_check_data)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=self.retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher timeout=timeout, retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher retry=retry)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise result
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher block_device_info)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher raise exception.InvalidLocalStorage(reason=reason, path=source)
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.052 TRACE oslo_messaging.rpc.dispatcher
2015-09-08 12:09:17.055 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] Returning exception cld5b12 is not on local storage: Block migration can not be used with shared storage.
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
    executor_callback))

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
    executor_callback)

  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
    result = func(ctxt, **new_args)

  File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
    payload)

  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
    return f(self, context, *args, **kw)

  File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function
    kwargs['instance'], e, sys.exc_info())

  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source
    block_device_info)

  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source
    raise exception.InvalidLocalStorage(reason=reason, path=source)

InvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.
 to caller
2015-09-08 12:09:17.056 ERROR oslo_messaging._drivers.common [req-2f320d08-ce60-4f5c-bfa3-c044246a3a18 admin admin] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n', ' File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/opt/stack/nova/nova/compute/manager.py", line 4899, in check_can_live_migrate_destination\n dest_check_data)\n', ' File "/opt/stack/nova/nova/compute/rpcapi.py", line 392, in check_can_live_migrate_source\n dest_check_data=dest_check_data)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call\n retry=self.retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send\n timeout=timeout, retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send\n retry=retry)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send\n raise result\n', 'InvalidLocalStorage_Remote: cld5b12 is not on local storage: Block migration can not be used with shared storage.\nTraceback (most recent call last):\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply\n executor_callback))\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch\n executor_callback)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch\n result = func(ctxt, **new_args)\n\n File "/opt/stack/nova/nova/exception.py", line 89, in wrapped\n payload)\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/exception.py", line 72, in wrapped\n return f(self, context, *args, **kw)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 399, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 377, in decorated_function\n kwargs[\'instance\'], e, sys.exc_info())\n\n File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 365, in decorated_function\n return function(self, context, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/manager.py", line 4929, in check_can_live_migrate_source\n block_device_info)\n\n File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5169, in check_can_live_migrate_source\n raise exception.InvalidLocalStorage(reason=reason, path=source)\n\nInvalidLocalStorage: cld5b12 is not on local storage: Block migration can not be used with shared storage.\n\n']

This check appears to only check the source, and is not actually testing to see if the storage is shared between the two nodes.

See original description
Revision history for this message
Sean Severson (sseverson) wrote :

On further review, block-migrate doesn't appear to be working at all. I restacked just the controller node and the unshared node and tried this test again with the same result. There is no shared storage between these two nodes at all, but n-cpu continues to report that it's detecting it.

description: updated
Sean Severson (sseverson)
description: updated
Revision history for this message
Sean Severson (sseverson) wrote :
Download full text (7.3 KiB)

Retested this afternoon from scratch. Two nodes with absolutely no shared storage between them will trigger the exact same exception. I also tried migrating instances in either direction, just to make sure it wasn't just an issue with the controller node.

In every case Nova claims that the source node is on shared storage and throws the exception.

Here's another log snippet from the most recent attempt:

2015-09-09 15:31:23.632 DEBUG keystoneclient.session [req-855df685-1203-4481-9a65-558408337709 admin admin] RESP: [200] content-length: 262 x-compute-request-id: req-9c71a22d-6d21-46e5-bf29-fe000910645d connection: keep-alive date: Wed, 09 Sep 2015 22:27:27 GMT content-type: application/json x-openstack-request-id: req-9c71a22d-6d21-46e5-bf29-fe000910645d
RESP BODY: {"connection_info": {"driver_volume_type": "iscsi", "data": {"target_discovered": true, "encrypted": false, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:20210002ac002db6", "target_portal": "172.16.1.24:3260", "target_lun": 0, "access_mode": "rw"}}}
 from (pid=14936) _http_log_response /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:215
2015-09-09 15:31:23.675 DEBUG nova.virt.libvirt.driver [req-855df685-1203-4481-9a65-558408337709 admin admin] skipping disk /dev/disk/by-path/ip-172.16.1.24:3260-iscsi-iqn.2000-05.com.3pardata:20210002ac002db6-lun-0 (vda) as it is a volume from (pid=14936) _get_instance_disk_info /opt/stack/nova/nova/virt/libvirt/driver.py:6476
2015-09-09 15:31:23.684 DEBUG nova.virt.libvirt.driver [req-855df685-1203-4481-9a65-558408337709 admin admin] skipping disk /dev/disk/by-path/ip-172.16.1.24:3260-iscsi-iqn.2000-05.com.3pardata:20210002ac002db6-lun-0 (vda) as it is a volume from (pid=14936) _get_instance_disk_info /opt/stack/nova/nova/virt/libvirt/driver.py:6476
2015-09-09 15:31:23.756 ERROR oslo_messaging.rpc.dispatcher [req-855df685-1203-4481-9a65-558408337709 admin admin] Exception during message handling: cld5b15 is not on local storage: Block migration can not be used with shared storage.
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher executor_callback))
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher executor_callback)
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher payload)
2015-09-09 15:31:23.756 TRACE oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7...

Read more...

Matt Riedemann (mriedem)
tags: added: migration volumes
Revision history for this message
Matt Riedemann (mriedem) wrote :

In comment 2 you say that block migration doesn't work at all, but we have a multi-node gate job with block migration that uses lvm and it doesn't have this issue, so is this is only an issue with NFS as the backend?

Revision history for this message
Matt Riedemann (mriedem) wrote :

Try this again with an LVM setup and see what happens. Also attach your nova and cinder config files. I'm wondering if stacking with NFS sets some config flags that makes nova assume shared storage.

Revision history for this message
Sean Severson (sseverson) wrote :

Attaching config files for nova and cinder.

Revision history for this message
Sean Severson (sseverson) wrote :
Revision history for this message
Sean Severson (sseverson) wrote :

Newest tests: Tried again with a LeftHand volume on the same Nova flavor with the same failure. Unstacked and reconfigured to add LVM and tested using an LVM volume on the same Nova flavor and again ended up with the same failure.

Revision history for this message
Walt Boring (walter-boring) wrote :

A clarification here.

The case here is when a nova instance is booted from a cinder volume and the nova flavor chosen has no local storage defined.

It looks like there is a specific check for that case here:
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5222-L5226

Which results in the exception being raised.

So, if this isn't supported, we should probably have the log message updated for this particular case and say it's not supported.

Revision history for this message
Sean Severson (sseverson) wrote :

As an FYI, this appears to be the change that caused this scenario to no longer work as I'd expected:

https://review.openstack.org/#/c/91722/37

Revision history for this message
Sean Severson (sseverson) wrote :

Another test scenario attempted:

Instance booted from a 3par volume. The instance uses a flavour with a local disk (Small, in this case) of 20 GB.

Attempted to migrate using the same environment as the last test (no shared storage for the instances directory) and the migration once again failed.

Revision history for this message
Matt Riedemann (mriedem) wrote :

I remember that https://review.openstack.org/#/c/91722/ introduced some upgrade issue in Kilo such that you couldn't upgrade from juno to kilo in some cases, I can't remember what exactly now, dansmith would remember since I think he fixed it.

https://review.openstack.org/#/c/108468/ was a related change after https://review.openstack.org/#/c/91722/ because it broke live migration with NFS.

Revision history for this message
Matt Riedemann (mriedem) wrote :

Looks like this is the change in the libvirt driver in liberty which is blowing up:

https://review.openstack.org/#/c/195885/

But it looks like that was specifically added for volume-backed live migration (boot from volume) but apparently doesn't work in this case?

Revision history for this message
Matt Riedemann (mriedem) wrote :

There is a tempest change here which is testing volume-backed live migration:

https://review.openstack.org/#/c/165233/

That was passing tests, so what's different about that test and the scenario here? The flavor?

Matt Riedemann (mriedem)
tags: added: liberty-rc-potential
Revision history for this message
Sean Severson (sseverson) wrote :

The commit message for that Tempest test specifically states that it's not to be used with block-migrate, which is what my test case is trying to use.

Revision history for this message
Matt Riedemann (mriedem) wrote :

I noticed the tempest config block_migrate_cinder_iscsi:

https://github.com/openstack/tempest/blob/master/tempest/config.py#L363

    cfg.BoolOpt('block_migrate_cinder_iscsi',
                default=False,
                help="Does the test environment block migration support "
                "cinder iSCSI volumes. Note, libvirt doesn't support this, "
                "see https://bugs.launchpad.net/nova/+bug/1398999"),

which calls out bug 1398999. Given https://review.openstack.org/#/c/139085/ - it seems that what you're trying to do here is maybe invalid given that bug?

Revision history for this message
Matt Riedemann (mriedem) wrote :

This also looks related, landed in Kilo: https://review.openstack.org/#/c/135074/

Matt Riedemann (mriedem)
tags: added: live-migration
John Garbutt (johngarbutt)
Changed in nova:
importance: Undecided → High
importance: High → Undecided
Revision history for this message
Matt Riedemann (mriedem) wrote :

Marking as liberty-backport-potential. It sounds like there might be a regression here but not really sure, would need someone to confirm and explain why. For now, we aren't going to hold up liberty rc1 for this.

tags: added: liberty-backport-potential
removed: liberty-rc-potential
Revision history for this message
Pawel Koniszewski (pawel-koniszewski) wrote :

It looks like your volume is attached to a VM through iSCSI:

RESP BODY: {"connection_info": {"driver_volume_type": "iscsi", "data": {"target_discovered": true, "encrypted": false, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:20210002ac002db6", "target_portal": "172.16.1.24:3260", "target_lun": 0, "access_mode": "rw"}}}

So it means that you can't block live migrate it - basically this happens because volume is "shared" through iSCSI (it also might be fiber channel in case of 3par backend).

Revision history for this message
Pawel Koniszewski (pawel-koniszewski) wrote :

Could you please clarify what do you use to attach your LVM/LeftHand/3par volumes? iSCSI, FC or something else? Marking as incomplete - we need clear steps for reproduction.

Changed in nova:
status: New → Incomplete
tags: added: live-migrate
removed: live-migration migration
Paul Murray (pmurray)
tags: added: live-migration
removed: live-migrate
lvmxh (shaohef)
Changed in nova:
assignee: nobody → lvmxh (shaohef)
lvmxh (shaohef)
Changed in nova:
assignee: lvmxh (shaohef) → nobody
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

Changed in nova:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.