Host field set too early during builds

Bug #1060255 reported by Brian Elliott
22
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Brian Elliott
Folsom
Fix Released
High
Vish Ishaya
nova (Ubuntu)
Fix Released
Undecided
Unassigned
Quantal
Fix Released
Undecided
Unassigned

Bug Description

The 'host' field on the instance is getting set in the scheduler during builds. This is too early. The compute host selected by scheduler needs to maintain final say over whether it has available resources to accept or deny incoming new build requests. Only after being accepted by a particular compute host should the 'host' field get set. If it is set too early, it will be erroneously included in resource calculations for that compute host and circumvent the resource tracker mechanism.

Brian Elliott (belliott)
Changed in nova:
assignee: nobody → Brian Elliott (belliott)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/14174

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/14174
Committed: http://github.com/openstack/nova/commit/5fd7a9dba127bae812333196a5fa48a933212aeb
Submitter: Jenkins
Branch: master

commit 5fd7a9dba127bae812333196a5fa48a933212aeb
Author: Brian Elliott <email address hidden>
Date: Mon Oct 8 18:35:14 2012 +0000

    Set instance host field after resource claim

    Set the 'host' field on the instance after the resource tracker
    on the compute node has accepted the build.

    The field is set after resources are confirmed to be available while the
    COMPUTE_RESOURCES_SEMAPHORE is held. The semaphore ensures the
    resources usage values will be consistent even if the
    update_available_resource periodic task audit runs.

    bug 1060255

    Change-Id: I92105ec14924960ac8ef7ca8c810783085314e10

Changed in nova:
status: In Progress → Fix Committed
Changed in nova:
importance: Undecided → Medium
tags: added: folsom-backport-potential
Revision history for this message
John Garbutt (johngarbutt) wrote :

This seems to be breaking nova-network multi-host:
https://bugs.launchpad.net/nova/+bug/1065004

Revision history for this message
Vish Ishaya (vishvananda) wrote :

Note to backporters, the dependent fix in https://bugs.launchpad.net/nova/+bug/1065004 should be backported along with this fix.

Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-1
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/16677

tags: removed: folsom-backport-potential
Michael Still (mikal)
tags: added: ops
Revision history for this message
Vish Ishaya (vishvananda) wrote :

fyi, this bug is a bit nastier than it looks: see the duplicate bug: https://bugs.launchpad.net/nova/+bug/1055069

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/16677
Committed: http://github.com/openstack/nova/commit/284f6ea64b0837981a24251825b19abef354a9a8
Submitter: Jenkins
Branch: stable/folsom

commit 284f6ea64b0837981a24251825b19abef354a9a8
Author: Brian Elliott <email address hidden>
Date: Mon Oct 8 18:35:14 2012 +0000

    Set instance host field after resource claim

    Set the 'host' field on the instance after the resource tracker
    on the compute node has accepted the build.

    The field is set after resources are confirmed to be available while the
    COMPUTE_RESOURCES_SEMAPHORE is held. The semaphore ensures the
    resources usage values will be consistent even if the
    update_available_resource periodic task audit runs.

    bug 1060255

    (cherry picked from commit 5fd7a9dba127bae812333196a5fa48a933212aeb)

    Also includes the following bugfix to the original patch:

    Set host prior to allocating network information.

    Make sure 'host' field on the instance is set before allocating
    network information.

    bug 1065004

    (cherry picked from commit 2649f14673f8ef5ca257583f1cdf5fe57d4734b9)
    Change-Id: I92105ec14924960ac8ef7ca8c810783085314e10

Changed in nova (Ubuntu):
status: New → Fix Released
Changed in nova (Ubuntu Quantal):
status: New → Confirmed
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Please test proposed package

Hello Brian, or anyone else affected,

Accepted nova into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/2012.2.1+stable-20121212-a99a802e-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nova (Ubuntu Quantal):
status: Confirmed → Fix Committed
tags: added: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (8.3 KiB)

This bug was fixed in the package nova - 2012.2.1+stable-20121212-a99a802e-0ubuntu1

---------------
nova (2012.2.1+stable-20121212-a99a802e-0ubuntu1) quantal-proposed; urgency=low

  * Ubuntu updates:
    - debian/control: Ensure novaclient is upgraded with nova,
      require python-keystoneclient >= 1:2.9.0. (LP: #1073289)
    - d/p/avoid_setuptools_git_dependency.patch: Refresh.
  * Dropped patches, applied upstream:
    - debian/patches/CVE-2012-5625.patch: [a99a802]
  * Resynchronize with stable/folsom (b55014ca) (LP: #1085255):
    - [a99a802] create_lvm_image allocates dirty blocks (LP: #1070539)
    - [670b388] RPC exchange name defaults to 'openstack' (LP: #1083944)
    - [3ede373] disassociate_floating_ip with multi_host=True fails
      (LP: #1074437)
    - [22d7c3b] libvirt imagecache should handle shared image storage
      (LP: #1075018)
    - [e787786] Detached and deleted RBD volumes remain associated with insance
      (LP: #1083818)
    - [9265eb0] live_migration missing migrate_data parameter in Hyper-V driver
      (LP: #1066513)
    - [3d99848] use_single_default_gateway does not function correctly
      (LP: #1075859)
    - [65a2d0a] resize does not migrate DHCP host information (LP: #1065440)
    - [102c76b] Nova backup image fails (LP: #1065053)
    - [48a3521] Fix config-file overrides for nova-dhcpbridge
    - [69663ee] Cloudpipe in Folsom: no such option: cnt_vpn_clients
      (LP: #1069573)
    - [6e47cc8] DisassociateAddress can cause Internal Server Error
      (LP: #1080406)
    - [22c3d7b] API calls to dis-associate an auto-assigned floating IP should
      return proper warning (LP: #1061499)
    - [bd11d15] libvirt: if exception raised during volume_detach, volume state
      is inconsistent (LP: #1057756)
    - [dcb59c3] admin can't describe all images in ec2 api (LP: #1070138)
    - [78de622] Incorrect Exception raised during Create server when metadata
      over 255 characters (LP: #1004007)
    - [c313de4] Fixed IP isn't released before updating DHCP host file
      (LP: #1078718)
    - [f4ab42d] Enabling Return Reservation ID with XML create server request
      returns no body (LP: #1061124)
    - [3db2a38] 'BackupCreate' should accept rotation parameter greater than or
      equal to zero (LP: #1071168)
    - [f7e5dde] libvirt reboot sometimes fails to reattach volumes
      (LP: #1073720)
    - [ff776d4] libvirt: detaching volume may fail while terminating other
      instances on the same host concurrently (LP: #1060836)
    - [85a8bc2] Used instance uuid rather than id in remove-fixed-ip
    - [42a85c0] Fix error on invalid delete_on_termination value
    - [6a17579] xenapi migrations fail w/ swap (LP: #1064083)
    - [97649b8] attach-time field for volumes is not updated for detach volume
      (LP: #1056122)
    - [8f6a718] libvirt: rebuild is not using kernel and ramdisk associated with
      the new image (LP: #1060925)
    - [fbe835f] live-migration and volume host assignement (LP: #1066887)
    - [c2a9150] typo prevents volume_tmp_dir flag from working (LP: #1071536)
    - [93efa21] Instances deleted during spawn leak network allocations
      (LP: #1068716)
    - [ebabd02] After restartin...

Read more...

Changed in nova (Ubuntu Quantal):
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.