Block live migration fails in packstack deployment

[Gabriel Szasz]

Description of problem:

Block live migration in Packstack deployment does not work without workaround.

How reproducible:
100%

Steps to Reproduce:
1. Deploy RDO in three-node setup (1 controller / 2 compute nodes) by Packstack on RHEL 7

      # packstack --install-hosts=${controller_ip},${node1_ip},${node2_ip} \
          --nagios-install=n \
          --os-ceilometer-install=n \
          --os-neutron-install=n \
          --novanetwork-pubif=${controller_if} \
          --novanetwork-privif=${controller_if} \
          --novacompute-privif=${node1_if} \
          --keystone-admin-passwd=redhat \
          --keystone-demo-passwd=redhat \
          --ssh-public-key=/root/.ssh/id_rsa.pub

2. Convince yourself that both compute nodes have the same CPUs and are connected to the network with the same network interfaces.

3. Add following line to /etc/nova/nova.conf on both compute nodes:

live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE

4. Restart openstack-nova-compute service on both compute nodes.

5. Run following commands on controller:

    # source keystonerc_admin
    # nova keypair-add --pub-key ~/.ssh/id_rsa root
    # nova boot --flavor m1.small --image cirros test --key_name root

6. Check the compute node on which was the instance spawned. In the next step use the other compute node as target host.

7. Run command:

    # nova live-migration --block-migrate test ${target_node}

Actual results:
CLI displays no error, but the instance is not migrated. Following error message appears in /var/log/nova-compute.log on the source compute node.

2014-07-08 16:59:23.894 28674 ERROR nova.virt.libvirt.driver [-] [instance: 61fe562c-7879-49c8-84b9-b100445414aa] Live Migration failure: operation failed: Failed to connect to remote libvirt URI qemu+ssh://<email address hidden>/system?no_verify=1&keyfile=/etc/nova/ssh/nova_migration_key: packet 1416128879 bytes received from server too large, want 16777216

Expected results:
Instance is migrated to the target host. No errors appear in log

Additional info:

The actual workaround is to replace /sbin/nologin shell for nova user on target node by /bin/bash.

The issue is related to the configuration made by Packstack.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vladan Popovic (<email address hidden>) on branch: master
Review: https://review.openstack.org/106436
Reason: moving to nova::compute::libvirt class in puppet-nova

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/106436
Committed: https://git.openstack.org/cgit/stackforge/packstack/commit/?id=34fe9b24d813d410ac34e021f71067190c52a1bd
Submitter: Jenkins
Branch: master

commit 34fe9b24d813d410ac34e021f71067190c52a1bd
Author: Vladan Popovic <email address hidden>
Date: Mon Jul 14 12:48:17 2014 +0200

    Enable migration support in libvirt

    QEMU breaks on ssh connections when the nova user has no login shell.
    Since the bug was discovered during live migration, this log snippet
    gives the error retrieved from nova:

    2014-07-08 16:59:23.894 28674 ERROR nova.virt.libvirt.driver [-]
    [instance: 61fe562c-7879-49c8-84b9-b100445414aa] Live Migration failure:
    operation failed: Failed to connect to remote libvirt URI
    qemu+ssh://nova@compute1/system?no_verify=1&keyfile=/etc/nova/ssh/nova_migration_key:
    packet 1416128879 bytes received from server too large, want 16777216

    This patch enables live migration when the nova user is set-up with
    /sbin/nologin by enabling migration support in nova::compute::libvirt.
    The puppet module sets the listen_tcp option to 1 whe migration is
    enabled.

    Closes bug 1340833

    Change-Id: Ic78e631e1099dcc0c500eef2998b8a6c4f5d5f7b

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: icehouse
Review: https://review.openstack.org/108381

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/108381
Committed: https://git.openstack.org/cgit/stackforge/packstack/commit/?id=fff859522a9049978a6127e9895ea52f5426aa6c
Submitter: Jenkins
Branch: icehouse

commit fff859522a9049978a6127e9895ea52f5426aa6c
Author: Vladan Popovic <email address hidden>
Date: Mon Jul 14 12:48:17 2014 +0200

    Enable migration support in libvirt

    QEMU breaks on ssh connections when the nova user has no login shell.
    Since the bug was discovered during live migration, this log snippet
    gives the error retrieved from nova:

    2014-07-08 16:59:23.894 28674 ERROR nova.virt.libvirt.driver [-]
    [instance: 61fe562c-7879-49c8-84b9-b100445414aa] Live Migration failure:
    operation failed: Failed to connect to remote libvirt URI
    qemu+ssh://nova@compute1/system?no_verify=1&keyfile=/etc/nova/ssh/nova_migration_key:
    packet 1416128879 bytes received from server too large, want 16777216

    This patch enables live migration when the nova user is set-up with
    /sbin/nologin by enabling migration support in nova::compute::libvirt.
    The puppet module sets the listen_tcp option to 1 whe migration is
    enabled.

    Closes bug 1340833

    Change-Id: Ic78e631e1099dcc0c500eef2998b8a6c4f5d5f7b
    (cherry picked from commit 34fe9b24d813d410ac34e021f71067190c52a1bd)

[Ivan Chavero (imcsk8) (ichavero-ichavero)]

The status of this bug report was changed automatically.