resize does not migrate DHCP host information

Bug #1065440 reported by Jian Wen
26
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Jian Wen
Folsom
Fix Released
Medium
Mark McLoughlin
nova (Ubuntu)
Fix Released
Undecided
Unassigned
Quantal
Fix Released
Undecided
Unassigned

Bug Description

In multi_host and multi-compute-node mode with allow_resize_to_same_host=false

How to reproduce:

1. boot an instance
     Its name is vm-26486
     Its host is dev2.
     On compute node dev2:
         $ cat /opt/stack/nova/networks/nova-br100.conf
         fa:16:3e:5d:65:44,vm-26486.novalocal,10.0.0.17

    On compute node dev0:
         $ cat /opt/stack/nova/networks/nova-br100.conf
          # empty

2. resize and resize-confirm it
     instance is migrated to compute node dev0
     On compute node dev0:
         $ cat /opt/stack/nova/networks/nova-br100.conf
          # empty

     On compute node dev2:
         $ cat /opt/stack/nova/networks/nova-br100.conf
         fa:16:3e:5d:65:44,vm-26486.novalocal,10.0.0.17

     Expected:
     On compute node dev0:
         $ cat /opt/stack/nova/networks/nova-br100.conf
          fa:16:3e:5d:65:44,vm-26486.novalocal,10.0.0.17

     If an instance is created on compute node dev2 now, vm-26486's DHCP info would be totally gone.
     That is :
     On compute node dev0:
         $ cat /opt/stack/nova/networks/nova-br100.conf
          # empty

     On compute node dev2:
         $ cat /opt/stack/nova/networks/nova-br100.conf
          # empty

     If on dev0 /opt/stack/nova/networks/nova-br100.conf is not updated in a certain time, vm-2648 will lease no IP.

Jian Wen (wenjianhn)
Changed in nova:
assignee: nobody → Jian Wen (wenjianhn)
Revision history for this message
Jian Wen (wenjianhn) wrote :

The reason is :

network_api.setup_networks_on_host is called before _instance_update(host=migration_ref['dest_compute']) in compute.manager._finish_resize, thus db.network_get_associated_fixed_ips(host='dest_compute_name') will not return this
instance's DHCP host info in FlatDHCPManager._setup_networks_on_host().

Update instance's host to dest_compute_node_name before setup_networks_on_host would fix this.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/14382

Changed in nova:
status: New → In Progress
Revision history for this message
Vish Ishaya (vishvananda) wrote :

might be good to check if a similar fix is needed for live/block migration

Changed in nova:
importance: Undecided → Medium
tags: added: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/14382
Committed: http://github.com/openstack/nova/commit/9ea41fc80b46b0c5adbab3ea191b92583feb02e0
Submitter: Jenkins
Branch: master

commit 9ea41fc80b46b0c5adbab3ea191b92583feb02e0
Author: Jian Wen <email address hidden>
Date: Fri Oct 12 15:52:17 2012 +0800

    Migrate DHCP host info during resize

    Update instance host to migration_ref['dest_compute'] before
    update_dhcp to make its DHCP host info will be written in dhcp-hostsfile.

    Make similiar change in finish_revert_resize.

    Fixes bug 1065440

    Change-Id: I9d3e0ebf8512f98f2943999e7ad95bc58f2bf585

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Jian Wen (wenjianhn) wrote :

Hi, vish

I will check.

Revision history for this message
Jian Wen (wenjianhn) wrote :

live/block migration is fine

Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-1
status: Fix Committed → Fix Released
tags: removed: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/16714

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/16714
Committed: http://github.com/openstack/nova/commit/65a2d0a3ce869ffa80344fed94a54e92de57a0c2
Submitter: Jenkins
Branch: stable/folsom

commit 65a2d0a3ce869ffa80344fed94a54e92de57a0c2
Author: Jian Wen <email address hidden>
Date: Fri Oct 12 15:52:17 2012 +0800

    Migrate DHCP host info during resize

    Update instance host to migration_ref['dest_compute'] before
    update_dhcp to make its DHCP host info will be written in dhcp-hostsfile.

    Make similiar change in finish_revert_resize.

    Fixes bug 1065440

    Change-Id: I9d3e0ebf8512f98f2943999e7ad95bc58f2bf585
    (cherry picked from commit 9ea41fc80b46b0c5adbab3ea191b92583feb02e0)

Changed in nova (Ubuntu):
status: New → Fix Released
Changed in nova (Ubuntu Quantal):
status: New → Confirmed
Revision history for this message
Matt Hilt (mjhilt-x) wrote :

I just ran into this issue on Essex/Precise. Is there hope for back port of this fix?

Revision history for this message
Jian Wen (wenjianhn) wrote :

No. I don't think so.
Many bugs(10+) has been fixed after Essex. Backporting is painful.

Revision history for this message
Clint Byrum (clint-fewbar) wrote : Please test proposed package

Hello Jian, or anyone else affected,

Accepted nova into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/2012.2.1+stable-20121212-a99a802e-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nova (Ubuntu Quantal):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Jian Wen (wenjianhn) wrote :

I will do it on Monday next week.

Revision history for this message
Riki Arslan (riki-arslan) wrote :

I can confirm that the package fixes the issue for Resize.

But, live/block migration does not migrate DHCP host information. We need a fix for that one as well.

Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (8.3 KiB)

This bug was fixed in the package nova - 2012.2.1+stable-20121212-a99a802e-0ubuntu1

---------------
nova (2012.2.1+stable-20121212-a99a802e-0ubuntu1) quantal-proposed; urgency=low

  * Ubuntu updates:
    - debian/control: Ensure novaclient is upgraded with nova,
      require python-keystoneclient >= 1:2.9.0. (LP: #1073289)
    - d/p/avoid_setuptools_git_dependency.patch: Refresh.
  * Dropped patches, applied upstream:
    - debian/patches/CVE-2012-5625.patch: [a99a802]
  * Resynchronize with stable/folsom (b55014ca) (LP: #1085255):
    - [a99a802] create_lvm_image allocates dirty blocks (LP: #1070539)
    - [670b388] RPC exchange name defaults to 'openstack' (LP: #1083944)
    - [3ede373] disassociate_floating_ip with multi_host=True fails
      (LP: #1074437)
    - [22d7c3b] libvirt imagecache should handle shared image storage
      (LP: #1075018)
    - [e787786] Detached and deleted RBD volumes remain associated with insance
      (LP: #1083818)
    - [9265eb0] live_migration missing migrate_data parameter in Hyper-V driver
      (LP: #1066513)
    - [3d99848] use_single_default_gateway does not function correctly
      (LP: #1075859)
    - [65a2d0a] resize does not migrate DHCP host information (LP: #1065440)
    - [102c76b] Nova backup image fails (LP: #1065053)
    - [48a3521] Fix config-file overrides for nova-dhcpbridge
    - [69663ee] Cloudpipe in Folsom: no such option: cnt_vpn_clients
      (LP: #1069573)
    - [6e47cc8] DisassociateAddress can cause Internal Server Error
      (LP: #1080406)
    - [22c3d7b] API calls to dis-associate an auto-assigned floating IP should
      return proper warning (LP: #1061499)
    - [bd11d15] libvirt: if exception raised during volume_detach, volume state
      is inconsistent (LP: #1057756)
    - [dcb59c3] admin can't describe all images in ec2 api (LP: #1070138)
    - [78de622] Incorrect Exception raised during Create server when metadata
      over 255 characters (LP: #1004007)
    - [c313de4] Fixed IP isn't released before updating DHCP host file
      (LP: #1078718)
    - [f4ab42d] Enabling Return Reservation ID with XML create server request
      returns no body (LP: #1061124)
    - [3db2a38] 'BackupCreate' should accept rotation parameter greater than or
      equal to zero (LP: #1071168)
    - [f7e5dde] libvirt reboot sometimes fails to reattach volumes
      (LP: #1073720)
    - [ff776d4] libvirt: detaching volume may fail while terminating other
      instances on the same host concurrently (LP: #1060836)
    - [85a8bc2] Used instance uuid rather than id in remove-fixed-ip
    - [42a85c0] Fix error on invalid delete_on_termination value
    - [6a17579] xenapi migrations fail w/ swap (LP: #1064083)
    - [97649b8] attach-time field for volumes is not updated for detach volume
      (LP: #1056122)
    - [8f6a718] libvirt: rebuild is not using kernel and ramdisk associated with
      the new image (LP: #1060925)
    - [fbe835f] live-migration and volume host assignement (LP: #1066887)
    - [c2a9150] typo prevents volume_tmp_dir flag from working (LP: #1071536)
    - [93efa21] Instances deleted during spawn leak network allocations
      (LP: #1068716)
    - [ebabd02] After restartin...

Read more...

Changed in nova (Ubuntu Quantal):
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.