floating ip does not move with live migration with multi_host

Bug #966529 reported by Travis Rhoden
34
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Jian Wen

Bug Description

I have a 2 compute-node setup running multi_host. If I assign a floating IP address to a VM and then migrate that VM to another compute host, the iptables NAT for that floating IP and the assignment of the IP to the physical network interface remain on the original host. I would expect these to move with the live migration.

The networking entries from my nova.conf:

# Networking
--network_manager=nova.network.manager.FlatDHCPManager
--force_dhcp_release
--fixed_range=10.55.100.0/25
--flat_network_dhcp_start=10.55.100.2
--flat_interface=eth2.100
--flat_network_bridge=br100
--libvirt_use_virtio_for_bridges
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--allow_same_net_traffic
--public_interface=eth1
--multi_host

If I have a VM starting on node2 (fixed ip 10.55.100.2), I see this interface assignment for the floating IP (10.54.100.225):
eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:30:48:f3:49:5d brd ff:ff:ff:ff:ff:ff
    inet 10.54.0.2/16 scope global eth1
    inet 10.54.100.225/32 scope global eth1

and iptables NAT entries (seen from iptables-save):
-A nova-network-OUTPUT -d 10.54.100.225/32 -j DNAT --to-destination 10.55.100.2
-A nova-network-PREROUTING -d 10.54.100.225/32 -j DNAT --to-destination 10.55.100.2
-A nova-network-float-snat -s 10.55.100.2/32 -j SNAT --to-source 10.54.100.225

All those entries look good/correct to me. Then I do a live-migration from node2 to node1. After that, I verify that the VM has moved successfully (it has) and I can reach it through the fixed network (I can). However, when I look at the floating network, all the entries from above are still present on node2! Here's node2 output from "ip a" again:

eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:30:48:f3:49:5d brd ff:ff:ff:ff:ff:ff
    inet 10.54.0.2/16 scope global eth1
    inet 10.54.100.225/32 scope global eth1

And then from node1:
eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:30:48:f3:49:5f brd ff:ff:ff:ff:ff:ff
    inet 10.54.0.1/16 scope global eth1

I definitely would have expected that IP address to change from eth1 on node2 to eth1 on node1, since I'm running multi_host. Consider that I may have done the live migration because I'm about to take node2 down for maintenance. Additionally, no NAT entries for that floating ip were made on node1. They remain on node2 as well. All the fixed ip entries were moved as expected.

Running on Ubuntu 12.04. Everything installed via apt-get. Latest packages as of 3/27.
node1:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu precise (development branch)
Release: 12.04
Codename: precise

node1:~# dpkg -l nova-compute
ii nova-compute 2012.1~rc1-0ubuntu2 OpenStack Compute - compute node
node1:~# dpkg -l nova-network
ii nova-network 2012.1~rc1-0ubuntu2 OpenStack Compute - Network manager

Revision history for this message
Vish Ishaya (vishvananda) wrote :

This is correct. We move the dhcp but not the floating ips.

Changed in nova:
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Vish Ishaya (vishvananda) wrote :

I don't think this will make it in to essex, so we will probably have to add something in the release notes letting people know that they will need to:
remove floating ip
migrate
add floating ip

Revision history for this message
Travis Rhoden (trhoden) wrote :

I can make do with the workaround from #2. +1 for release notes comments.

Thanks for the info.

Thierry Carrez (ttx)
tags: added: essex-release-notes
Jian Wen (wenjianhn)
Changed in nova:
assignee: nobody → Jian Wen (wenjianhn)
Jian Wen (wenjianhn)
Changed in nova:
assignee: Jian Wen (wenjianhn) → nobody
Revision history for this message
Alessandro Tagliapietra (tagliapietra.alessandro) wrote :

So this bug will be solved with that workaround?

Changed in nova:
importance: Medium → High
Jian Wen (wenjianhn)
Changed in nova:
assignee: nobody → Sina Web Service Dev (sws)
Jian Wen (wenjianhn)
Changed in nova:
assignee: Sina Web Service Dev (sws) → Jian Wen (wenjianhn)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/15996

Changed in nova:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/15996
Committed: http://github.com/openstack/nova/commit/3c173a744a5d4ae7845411865a7f85d9c2411e74
Submitter: Jenkins
Branch: master

commit 3c173a744a5d4ae7845411865a7f85d9c2411e74
Author: Jian Wen <email address hidden>
Date: Wed Nov 14 01:22:40 2012 +0800

    Migrate floating ip addresses in multi_host live_migration

    Make migration of floating ip addresses work and work only
    in multi_host mode.
    Add test case for post_live_migration_at_destination.

    Fixes bug 966529 and bug 1029291

    Change-Id: Ia79030e5bcfc55e5a33b20a3d6eead01fa5b255a

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-1
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/25715

Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.