Live migration fails with ceph rbd attached block device between different nova-compute instances

[Peter Sabaini]

We have two nova-compute services with distinct names to support different types of hardware.

When live migrating instances with attached volumes across those services we run into authentication issues against ceph.

ERROR nova.virt.libvirt.driver [req-...] [instance: ...] Live Migration failure: internal error: process exited while connecting to monitor: 2016-07-20T08:16:15.614465Z qemu-system-x86_64: -drive file=rbd:cinder/volume-7ecfac34-afdb-4944-8670-154e5631671c:id=nova-compute:key=Abarkey==:auth_supported=cephx\;none:mon_host=1.1.1.1\:6789\;1.1.1.2\:6789\;1.1.1.3\:6789,format=raw,if=none,id=drive-virtio-disk1,serial=7ecfac34-afdb-4944-8670-154e5631671c,cache=none: error connecting

This is, afaict, due to the fact that the ceph charm sets up credentials per service, and considers those two nova-compute separate services:

$ sudo ceph auth list
...
client.compute-only
key: Afookey==
caps: [mon] allow rw
caps: [osd] allow rwx
...
client.nova-compute
key: Abarkey==
caps: [mon] allow rw
caps: [osd] allow rwx

Also cf. ceph charm, in hooks/ceph_hooks.py:L597, (client-relation-joined hook function). This function sets up client authentication per service, and determines service name by splitting up the peer unit name

[James Page (james-page)]

As the key is encoded in the data, this look odd to me:

"key=Abarkey==:auth_supported=cephx\;none:mon_host=1.1.1.1\:6789\;1.1.1.2\:6789\;1.1.1.3\:6789"

The key is normally encoded using a libvirt secret - I'm not 100% sure of the designed behaviour in libvirt for this particular use case (migrating between compute hosts with different keys stored in the ceph-key secret).

[James Page (james-page)]

I think having different keys is correct - we'll need to dig into the underlying migration code to figure out why this actually happens.

[James Page (james-page)]

@Peter

Please can you confirm which OpenStack release you are using as this will determine libvirt/qemu version as well.

Marking 'Incomplete' pending that information - please set back to 'New' once provided.

[Peter Sabaini (peter-sabaini)]

Thanks James -- this is Mitaka running on Trusty

ii libvirt-bin 1.3.1-1ubuntu10.1~cloud0
ii nova-compute 2:13.0.0-0ubuntu5~cloud0
ii qemu-system 1:2.5+dfsg-5ubuntu10.2~cloud0

[Billy Olsen (billy-olsen)]

I've looked into this previously. As Peter mentioned, when mounting a ceph rbd device, the ceph user is included in the volume attachment properties in the libvirt domain xml generated by nova. When this is transferred across to a target hypervisor, the ceph user information must exist on the target side as well.

The ceph-mon creates users based on the name of the remote application that connects. Thus, when two nova-compute applications are deployed with different names they will have unique ceph credentials to access the cluster. The live migration fails because of this.

To fix this, we should probably add a field that allows the remote service to specify a ceph user name. I've looked before and it was somewhat non-trivial to change this.