RBD + Nova migration fails because imports an existing disk

[gadLinux]

Hi I found that migration with rbd will never work if using ceph for shared storage.

Current driver tries to import the images to the new compute node but the images are already there because it's shared resource.

So it tries this:

 rbd import --pool volumes /var/lib/nova/instances/_base/swap_1024 b4c64eda-b684-4f3b-933c-95335ac202b8_disk.swap --new-format --id cinder --conf /etc/ceph/ceph.conf

But it fails because this disk already exists.

[Expected behavior]

It should not try to import disk, it should just use it.

Current error:

2014-07-25 22:53:13.134 4249 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 274, in decorated_function\n pass\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 327, in decorated_function\n function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 248, in decorated_function\n migration.instance_uuid, exc_info=True)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 235, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3559, in finish_resize\n self._set_instance_error_state(context, instance[\'uuid\'])\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3547, in finish_resize\n disk_info, image)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3518, in _finish_resize\n block_device_info, power_on)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5017, in finish_migration\n block_device_info=None, inject_files=False)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2680, in _create_image\n swap_mb=swap_mb)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 192, in cache\n *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 678, in create_image\n libvirt_utils.import_rbd_image(*args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 255, in import_rbd_image\n execute(\'rbd\', \'import\', *args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 53, in execute\n return utils.execute(*args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 164, in execute\n return processutils.execute(*cmd, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 193, in execute\n cmd=\' \'.join(cmd))\n', "ProcessExecutionError: Unexpected error while running command.\nCommand: rbd import --pool volumes /var/lib/nova/instances/_base/swap_1024 b4c64eda-b684-4f3b-933c-95335ac202b8_disk.swap --new-format --id cinder --conf /etc/ceph/ceph.conf\nExit code: 17\nStdout: ''\nStderr: 'rbd: image creation failed\\n\\rImporting image: 0% complete...failed.\\n2014-07-25 22:53:12.604111 7f5af3f877c0 -1 librbd: rbd image b4c64eda-b684-4f3b-933c-95335ac202b8_disk.swap already exists\\nrbd: import failed: (17) File exists\\n'\n"]

[Richard Arends (l-lauuchpad-s)]

I have this issue also:

Openstack Icehouse on Ubuntu 14.04, using the Ubuntu packages:

ceph 0.80.5-0ubuntu0.14.04.1
ceph-common 0.80.5-0ubuntu0.14.04.1
libcephfs1 0.80.5-0ubuntu0.14.04.1
libvirt-bin 1.2.2-0ubuntu13.1.2
libvirt0 1.2.2-0ubuntu13.1.2
nova-common 1:2014.1.2-0ubuntu1.1
nova-compute 1:2014.1.2-0ubuntu1.1
nova-compute-kvm 1:2014.1.2-0ubuntu1.1
nova-compute-libvirt 1:2014.1.2-0ubuntu1.1
python-ceph 0.80.5-0ubuntu0.14.04.1
python-libvirt 1.2.2-0ubuntu1
python-nova 1:2014.1.2-0ubuntu1.1
python-novaclient 1:2.17.0-0ubuntu1

Using this config settings in nova.conf:

libvirt_images_type=rbd
libvirt_images_rbd_pool=openstack_instances
libvirt_images_rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=cinder
rbd_secret_uuid=<secret uuid>
libvirt_inject_password=false
libvirt_inject_key=false
libvirt_inject_partition=-2

libvirt_live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST"

[Matt Jarvis (matt-jarvis)]

This also happens for us on resize.

[Jon Bernard (jbernard)]

Can you list the exact commands you're running to reproduce? I've been working on a few related issues recently and I'm happy to take a look.

[J-PMethot (jpmethot)]

I'm not related at all to the one who reported this bug, but I am able to reproduce simply by trying to resize a VM from Horizon. Since the resize process also does a migration, this bug affect the resize process during the migration step. Please note that I'm assuming this is the same bug because the error and symptoms are exactly the same.

That would mean provision vm -> open the "more" drop down menu -> click resize vm -> choose a bigger flavour -> look at the migration fail and the error show up in the logs.

[Jon Bernard (jbernard)]

Initially, at least on the latest github code, this appears to work. Has anyone confirmed this behaviour with the kilo-3 release?

[J-PMethot (jpmethot)]

We have recently built a juno setup and the issue appears to be fixed on juno 2014.2.2 . However, we are running into the revert resize bug https://bugs.launchpad.net/nova/+bug/1399244 . But I understand this is another matter entirely, which is fixed in kilo I believe.

[Daniel Marks (d3n14l)]

Any chance we get a fix for Icehouse?

[Sean McGinnis (sean-mcginnis)]

Is this still an issue in master?

[YaoZheng_ZTE (zheng-yao1)]

This bug has been fixed. After K version, nova migration is ok.

[Andrew (openstackdotby)]

it is, of course, endless process of fixing bugs not in current relelse but in future. but may be someone know set of patches cinder in metaka? I can't to make update of all cluster only due to this. Live system.