Ubuntu Server Natty: ganeti2 (2.1.6) migrate feature doesn't work with qemu-kvm

[javier_cepa]

Binary package hint: ganeti2

What distribution are you using?

Distributor ID: Ubuntu
Description: Ubuntu 11.04
Release: 11.04
Codename: natty

What steps will reproduce the problem?

gnt-instance migrate vmtest

What is the expected output? What do you see instead?

After distro upgrade migratie doesn't work. failover works perfectly.

root@drbd01:~# gnt-instance migrate -d vmtest
2011-05-20 13:32:44,425: gnt-instance migrate pid=2634 cli:1671 INFO run with arguments '-d vmtest.ss.local.com'

Instance vmtest.ss.local.com will be migrated. Note that

migration might impact the instance if anything goes wrong (e.g. due

to bugs in the hypervisor). Continue?

y/[n]/?: y

Fri May 20 13:32:46 2011 Migrating instance vmtest.ss.local.optenet.com
Fri May 20 13:32:46 2011 * checking disk consistency between source and target
Fri May 20 13:32:47 2011 * switching node drbd02.ss.local.com to secondary mode
Fri May 20 13:32:47 2011 * changing into standalone mode
Fri May 20 13:32:47 2011 * changing disks into dual-master mode
Fri May 20 13:32:48 2011 * wait until resync is done
Fri May 20 13:32:48 2011 * preparing drbd02.ss.local.com to accept the instance
Fri May 20 13:32:48 2011 * Comando KVM [\n [\n "/usr/bin/kvm",\n "-name",\n "vmtest.ss.local.com",\n "-m",\n 512,\n "-smp",\n 1,\n "-pidfile",\n "/var/run/ganeti/kvm-hypervisor/pid/vmtest.ss.local.com",\n "-daemonize",\n "-enable-kvm",\n "-drive",\n "file=/var/run/ganeti/instance-disks/vmtest.ss.local.com:0,format=raw,if=ide",\n "-boot",\n "d",\n "-drive",\n "file=/root/ISOS/WinZPSP3ES.iso,format=raw,media=cdrom",\n "-vnc",\n ":5128",\n "-usbdevice",\n "tablet",\n "-monitor",\n "unix:/var/run/ganeti/kvm-hypervisor/ctrl/vmtest.ss.local.com.monitor,server,nowait",\n "-serial",\n "unix:/var/run/ganeti/kvm-hypervisor/ctrl/vmtest.ss.local.com.serial,server,nowait"\n ],\n [\n {\n "mac": "aa:00:00:64:f2:cd",\n "nicparams": {\n "link": "br0",\n "mode": "bridged"\n }\n }\n ],\n {\n "acpi": true,\n "boot_order": "cdrom",\n "cdrom_image_path": "/root/ISOS/WinZPSP3ES.iso",\n "disk_cache": "default",\n "disk_type": "ide",\n "initrd_path": "",\n "kernel_args": "ro",\n "kernel_path": "",\n "kvm_flag": "enabled",\n "migration_bandwidth": 32,\n "migration_downtime": 30,\n "migration_port": 8102,\n "nic_type": "rtl8139",\n "root_path": "/dev/vda1",\n "security_domain": "",\n "security_model": "none",\n "serial_console": true,\n "usb_mouse": "",\n "use_chroot": false,\n "use_localtime": false,\n "vhost_net": false,\n "vnc_bind_address": "0.0.0.0",\n "vnc_password_file": "",\n "vnc_tls": false,\n "vnc_x509_path": "",\n "vnc_x509_verify": false\n }\n]\n
Fri May 20 13:32:48 2011 * migrating instance to drbd02.ss.local.com
Fri May 20 13:33:00 2011 * drbd01.ss.local.com {'network_port': 11028, 'serial_no': 2, 'name': 'vmtest.ss.local.com', 'hypervisor': 'kvm', 'disks': [{'logical_id': ('drbd01.ss.local.com', 'drbd02.ss.local.com', 11029, 3, 3, 'c001ed0374ac2b89e7a3d6b51bac098e8f05d0a3'), 'dev_type': 'drbd8', 'mode': 'rw', 'physical_id': ('10.0.1.2', 11029, '10.0.1.1', 11029, 3, 'c001ed0374ac2b89e7a3d6b51bac098e8f05d0a3'), 'children': [{'dev_type': 'lvm', 'logical_id': ('kvmvg', 'ad89192e-c940-4f09-ac86-be0216a39ad2.disk0_data'), 'physical_id': ('kvmvg', 'ad89192e-c940-4f09-ac86-be0216a39ad2.disk0_data'), 'size': 10240}, {'dev_type': 'lvm', 'logical_id': ('kvmvg', 'ad89192e-c940-4f09-ac86-be0216a39ad2.disk0_meta'), 'physical_id': ('kvmvg', 'ad89192e-c940-4f09-ac86-be0216a39ad2.disk0_meta'), 'size': 128}], 'iv_name': 'disk/0', 'size': 10240}], 'uuid': 'ce7a9357-71f1-4fab-91c7-e9f6829e64d7', 'primary_node': 'drbd01.ss.local.com', 'hvparams': {'nic_type': 'rtl8139', 'vnc_bind_address': '0.0.0.0', 'disk_type': 'ide'}, 'mtime': 1305891101.452351, 'disk_template': 'drbd', 'nics': [{'mac': 'aa:00:00:64:f2:cd', 'nicparams': {}}], 'admin_up': True, 'os': 'debootstrap+default', 'beparams': {'vcpus': 1, 'memory': 512}, 'ctime': 1305890719.470131} 10.0.1.2 True
Fri May 20 13:33:00 2011 Migration failed, aborting
Fri May 20 13:33:00 2011 * switching node drbd02.ss.local.com to secondary mode
Fri May 20 13:33:00 2011 * changing into standalone mode
Fri May 20 13:33:01 2011 * changing disks into single-master mode
Fri May 20 13:33:01 2011 * wait until resync is done
2011-05-20 13:33:01,896: gnt-instance migrate pid=2634 cli:1680 ERROR Error during command processing

Traceback (most recent call last):

  File "/usr/lib/pymodules/python2.7/ganeti/cli.py", line 1676, in GenericMain
    result = func(options, args)
  File "/usr/sbin/gnt-instance", line 890, in MigrateInstance
    SubmitOpCode(op, cl=cl, opts=opts)
  File "/usr/lib/pymodules/python2.7/ganeti/cli.py", line 1482, in SubmitOpCode
    op_results = PollJob(job_id, cl=cl, feedback_fn=feedback_fn)
  File "/usr/lib/pymodules/python2.7/ganeti/cli.py", line 1464, in PollJob
    return GenericPollJob(job_id, _LuxiJobPollCb(cl), reporter)
  File "/usr/lib/pymodules/python2.7/ganeti/cli.py", line 1299, in GenericPollJob
    errors.MaybeRaise(msg)
  File "/usr/lib/pymodules/python2.7/ganeti/errors.py", line 396, in MaybeRaise
    raise err_class, tuple(result[1])

OpExecError: Could not migrate instance vmtest.ss.local.com: Failed to migrate instance: Migration failed at the kvm level
Failure: command execution error:
Could not migrate instance vmtest.ss.local.com: Failed to migrate instance: Migration failed at the kvm level

Please provide any additional information below.

qemu-kvm_0.12.5+noroms-0ubuntu7.2_amd64.deb
 Works FINE
qemu-kvm_0.14.0+noroms-0ubuntu4_amd64.deb
 DO NOT WORK

root@drbd01:~# gnt-cluster version

Software version: 2.1.6
Internode protocol: 30
Configuration format: 2010000
OS api version: 15
Export interface: 0

********************
Comment 4 by project member <email address hidden>, May 20 (2 days ago)

Yes. Can you check 2.4.X, please? You can find debs in the debian repositories.

Thanks,

Guido

Comment 5 by <email address hidden>, May 20 (2 days ago) via email

Hi,

Packages used for upgrade:

ganeti2_2.4.1-1_all.deb
ganeti-htools_0.3.1-1_amd64.deb
libgmp10_5.0.1+dfsg-7_amd64.deb
python-crypto_2.3-1_amd64.deb
python-paramiko_1.7.6-6_all.deb

After upgrade and postupgrade task....

root@drbd01:~# gnt-instance migrate -d xpvu.ss.local.com
2011-05-21 00:54:46,807: gnt-instance migrate pid=3143 cli:1926 INFO run
with arguments '-d xpvu.ss.local.com'
Instance xpvu.ss.local.com will be migrated. Note that
migration might impact the instance if anything goes wrong (e.g. due
to bugs in the hypervisor). Continue?
y/[n]/?: y
Sat May 21 00:54:54 2011 Migrating instance xpvu.ss.local.com
Sat May 21 00:54:54 2011 * checking disk consistency between source and
target
Sat May 21 00:54:54 2011 * switching node drbd01.ss.local.com to secondary
mode
Sat May 21 00:54:55 2011 * changing into standalone mode
Sat May 21 00:54:55 2011 * changing disks into dual-master mode
Sat May 21 00:54:56 2011 * wait until resync is done
Sat May 21 00:54:56 2011 * preparing drbd01.ss.local.com to accept the
instance
Sat May 21 00:54:56 2011 * migrating instance to drbd01.ss.local.com
Sat May 21 00:55:40 2011 * switching node drbd02.ss.local.com to secondary
mode
Sat May 21 00:55:40 2011 * wait until resync is done
Sat May 21 00:55:40 2011 * changing into standalone mode
Sat May 21 00:55:40 2011 * changing disks into single-master mode
Sat May 21 00:55:41 2011 * wait until resync is done
Sat May 21 00:55:41 2011 * done

It works!

Comment 6 by project member <email address hidden>, Yesterday (17 hours ago)

Marking the issue as invalid, as it was fixed in a later version.

Thanks in advance,

http://code.google.com/p/ganeti/issues/detail?id=160&colspec=ID%20Type%20Component%20Status%20Priority%20Stars%20Opened%20Modified%20Milestone%20Owner%20Summary