check_can_live_migrate_source does not return data (xenapi)

[Armando Migliaccio]

I try to live block migrate on nova trunk with xenapi as virt driver and I see this stacktrace in the scheduler log:

2013-02-07 05:18:02.768 ERROR nova.openstack.common.rpc.amqp [req-33d5b197-0db2-48f3-b1f6-c2ddd60b0855 YWg4YWB8q9aDwKbdC4lR4DAY YWg4YWB8q9aDwKbdC4lR4DAY] Exception during message handling
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 99, in live_migration
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp context, ex, {})
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 94, in live_migration
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp block_migration, disk_over_commit)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/scheduler/driver.py", line 170, in schedule_live_migration
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp context, instance, dest, block_migration, disk_over_commit)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 225, in check_can_live_migrate_destination
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp ctxt, destination, None))
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 80, in call
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp return rpc.call(context, self._get_topic(topic), msg, timeout)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 109, in call
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 765, in call
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp rpc_amqp.get_connection_pool(conf, Connection))
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 379, in call
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp rv = list(rv)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 347, in __iter__
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp raise result
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp AttributeError: 'NoneType' object has no attribute 'update'
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 109, in wrapped
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2657, in check_can_live_migrate_destination
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp migrate_data.update(dest_check_data['migrate_data'])
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp
2013-02-07 05:18:02.768 10466 TRACE nova.openstack.common.rpc.amqp AttributeError: 'NoneType' object has no attribute 'update'

I narrowed it down to the fact that

https://github.com/openstack/nova/blob/master/nova/virt/xenapi/driver.py#L443

does not return dest_check_data.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/21449

[Mate Lakat (mate-lakat)]

Hi Armando,

I tried to reproduce the problem, and ran into another one.

nova boot --flavor m1.tiny --image d5f2be40-fc6a-439d-a0d2-fa8b0d23377e somemachine
nova show c0f06b42-fc5e-419a-a829-e7cffb3e45ac
...
OS-EXT-SRV-ATTR:host | DevStackComputeSlave
...
stack@DevStackOSDomU:~/devstack$ nova-manage service list
Binary Host Zone Status State Updated_At
nova-compute DevStackComputeSlave nova enabled :-) 2013-02-08 16:06:36
nova-network DevStackComputeSlave internal enabled :-) 2013-02-08 16:06:33
nova-compute DevStackOSDomU nova enabled :-) 2013-02-08 16:06:36
nova-network DevStackOSDomU internal enabled :-) 2013-02-08 16:06:35

nova live-migration c0f06b42-fc5e-419a-a829-e7cffb3e45ac DevStackOSDomU --block-migrate

2013-02-08 16:09:28.794 ERROR nova.compute [req-467ae35b-3446-415f-8ef8-3e2e411ff189 admin demo] No db access allowed in nova-compu
te: File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
    func(*args, **kwargs)
  File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)
  File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)
  File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
    return f(self, context, *args, **kw)
  File "/opt/stack/nova/nova/compute/manager.py", line 2732, in check_can_live_migrate_source
    None)
  File "/opt/stack/nova/nova/compute/api.py", line 2349, in is_volume_backed_instance
    bdms = bdms or self.get_instance_bdms(context, instance)
  File "/opt/stack/nova/nova/compute/api.py", line 2346, in get_instance_bdms
    instance['uuid'])
  File "/opt/stack/nova/nova/db/api.py", line 1095, in block_device_mapping_get_all_by_instance
    instance_uuid)
  File "/opt/stack/nova/bin/nova-compute", line 65, in __call__
    stacktrace = "".join(traceback.format_stack())

I am a bit confused, although it seems, that live block migration is broken.

[John Garbutt (johngarbutt)]

Well since nova-conductor was added, you nova-compute shouldn't be making any db calls, so that could well be something that hasn't been sorted yet!

[Mate Lakat (mate-lakat)]

I was surprised Armando haven't seen that message.

[Armando Migliaccio (armando-migliaccio)]

Mate, I am afraid I don't see that message; I believe compute calls go through nova-conductor. Are you running trunk? I am slightly behind.

[Andrew Laski (alaski)]

A code change went in right around when the patch for this was proposed which breaks the db access. I filed a bug for it at https://bugs.launchpad.net/nova/+bug/1122432. So while the patch addresses the reported issue there is now another one that also needs to be fixed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/21449
Committed: http://github.com/openstack/nova/commit/70c6882763116ef6515090f136fc0769698a6816
Submitter: Jenkins
Branch: master

commit 70c6882763116ef6515090f136fc0769698a6816
Author: Armando Migliaccio <email address hidden>
Date: Thu Feb 7 18:45:46 2013 +0000

    Return dest_check_data as expected by the Scheduler

    the scheduler driver expects data in return when
    calling compute_rcp.check_can_live_migrate_destination
    in method 'schedule_live_migration'.

    In turn check_can_live_migrate_destination calls
    check_can_live_migrate_source that also expects data
    returned from the driver.

    Fixes bug #1118491

    Change-Id: I3dbfa716d5d9bc849c1f67005e7bb1b8ba1ccd30