Inappropriate exception handling on kvm live/block migration

Bug #917615 reported by Kei Masumoto
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Kei Masumoto
Diablo
Fix Released
Undecided
Kiall Mac Innes
Essex
Fix Released
Undecided
Unassigned
nova (Ubuntu)
Fix Released
Undecided
Unassigned
Precise
Fix Released
Undecided
Unassigned

Bug Description

1. Cannot check exception messages when live/block migration fails.
Error message is below.

> ---------------------------------------------------------------------
> 2012-01-17 05:19:10,250 DEBUG nova.rpc [-] Making asynchronous cast on
> compute.os-sv02... from (pid=18400) cast
> /opt/openstack/dev/nova/nova/rpc/impl_kombu.py:784
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 97, in wait
> readers.get(fileno, noop).cb(fileno)
> File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 192, in main
> result = function(*args, **kwargs)
> File "/opt/openstack/dev/nova/nova/virt/libvirt/connection.py", line 1717, in _live_migration
> raise
> TypeError: exceptions must be old-style classes or derived from
> BaseException, not NoneType Removing descriptor: 9
> ---------------------------------------------------------------------

2. invalid argument
nova.compute.manager.recover_live_migration() should follow recent nova.virt.libvirt.connection.destroy() changes.
Otherwise, it cause an TypeError.

         self.driver.destroy(instance_ref, network_info,
- block_device_info, True)
+ block_device_info)

Related branches

Revision history for this message
Kei Masumoto (masumotok) wrote :
keiji niwa (niwakij)
Changed in nova:
assignee: nobody → NTT DATA (nttdata)
assignee: NTT DATA (nttdata) → nobody
Kei Masumoto (masumotok)
Changed in nova:
assignee: nobody → Kei Masumoto (masumotok)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/3134

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/3134
Committed: http://github.com/openstack/nova/commit/52d58ad583350f2448b3971eeb3f1df541cfbfe5
Submitter: Jenkins
Branch: master

commit 52d58ad583350f2448b3971eeb3f1df541cfbfe5
Author: masumotok <email address hidden>
Date: Wed Jan 18 19:29:31 2012 +0900

    Fix bug #917615

    Change-Id: I773b3f026fd7ebcb2aefcd2ef348cca0c5477238

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/diablo)

Fix proposed to branch: stable/diablo
Review: https://review.openstack.org/3232

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/diablo)

Reviewed: https://review.openstack.org/3232
Committed: http://github.com/openstack/nova/commit/4f43496502aef88752e616695e29f930f09009c3
Submitter: Jenkins
Branch: stable/diablo

commit 4f43496502aef88752e616695e29f930f09009c3
Author: Kiall Mac Innes <email address hidden>
Date: Fri Jan 20 20:26:24 2012 +0000

    Fix bug #917615.

    Partial Cherry-Pick of 52d58ad583350f2448b3971eeb3f1df541cfbfe5.

    Changes to nova.compute.manager.recover_live_migration() discarded as they are unnecessary for diablo.

    Conflicts:

     nova/compute/manager.py

    Change-Id: I773b3f026fd7ebcb2aefcd2ef348cca0c5477238

tags: added: in-stable-diablo
Thierry Carrez (ttx)
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-4 → 2012.1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/essex)

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/11813

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/essex)

Reviewed: https://review.openstack.org/11813
Committed: http://github.com/openstack/nova/commit/5d63601a8cd79d7d3311b32f6fe8c30fb2ec4580
Submitter: Jenkins
Branch: stable/essex

commit 5d63601a8cd79d7d3311b32f6fe8c30fb2ec4580
Author: Takaaki Suzuki <email address hidden>
Date: Wed Aug 8 11:21:55 2012 +0900

    Add error log for live migration.

    Fixes bug: 917615
    Change-Id: I740c0def3b1f759aec3d4ae5fe3264484c704807
    Cherry-picked: adb98d50d2af36fc6215cbec63fe89e3e67d8fa2
    Signed-off-by: Takaaki Suzuki <email address hidden>

tags: added: in-stable-essex
Dave Walker (davewalker)
Changed in nova (Ubuntu):
status: New → Fix Released
Changed in nova (Ubuntu Precise):
status: New → Confirmed
Revision history for this message
Adam Gandelman (gandelman-a) wrote : Verification report.

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Nova has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/3134
Stable review: https://review.openstack.org/11813

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Revision history for this message
Adam Gandelman (gandelman-a) wrote :

Test coverage log.

tags: added: verification-done
Revision history for this message
Launchpad Janitor (janitor) wrote :
Download full text (5.4 KiB)

This bug was fixed in the package nova - 2012.1.3+stable-20120827-4d2a4afe-0ubuntu1

---------------
nova (2012.1.3+stable-20120827-4d2a4afe-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot, fixes FTBFS in -proposed. (LP: #1041120)
  * Resynchronize with stable/essex (4d2a4afe):
    - [5d63601] Inappropriate exception handling on kvm live/block migration
      (LP: #917615)
    - [ae280ca] Deleted floating ips can cause instance delete to fail
      (LP: #1038266)

nova (2012.1.3+stable-20120824-86fb7362-0ubuntu1) precise-proposed; urgency=low

  * New upstream snapshot. (LP: #1041120)
  * Dropped, superseded by new snapshot:
    - debian/patches/CVE-2012-3447.patch: [d9577ce]
    - debian/patches/CVE-2012-3371.patch: [25f5bd3]
    - debian/patches/CVE-2012-3360+3361.patch: [b0feaff]
  * Resynchronize with stable/essex (86fb7362):
    - [86fb736] Libvirt driver reports incorrect error when volume-detach fails
      (LP: #1029463)
    - [272b98d] nova delete lxc-instance umounts the wrong rootfs (LP: #971621)
    - [09217ab] Block storage connections are NOT restored on system reboot
      (LP: #1036902)
    - [d9577ce] CVE-2012-3361 not fully addressed (LP: #1031311)
    - [e8ef050] pycrypto is unused and the existing code is potentially insecure
      to use (LP: #1033178)
    - [3b4ac31] cannot umount guestfs (LP: #1013689)
    - [f8255f3] qpid_heartbeat setting in ineffective (LP: #1030430)
    - [413c641] Deallocation of fixed IP occurs before security group refresh
      leading to potential security issue in error / race conditions
      (LP: #1021352)
    - [219c5ca] Race condition in network/deallocate_for_instance() leads to
      security issue (LP: #1021340)
    - [f2bc403] cleanup_file_locks does not remove stale sentinel files
      (LP: #1018586)
    - [4c7d671] Deleting Flavor currently in use by instance creates error
      (LP: #994935)
    - [7e88e39] nova testsuite errors on newer versions of python-boto (e.g.
      2.5.2) (LP: #1027984)
    - [80d3026] NoMoreFloatingIps: Zero floating ips available after repeatedly
      creating and destroying instances over time (LP: #1017418)
    - [4d74631] Launching with source groups under load produces lazy load error
      (LP: #1018721)
    - [08e5128] API 'v1.1/{tenant_id}/os-hosts' does not return a list of hosts
      (LP: #1014925)
    - [801b94a] Restarting nova-compute removes ip packet filters (LP: #1027105)
    - [f6d1f55] instance live migration should create virtual_size disk image
      (LP: #977007)
    - [4b89b4f] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [6e873bc] [nova][volumes] Exceeding volumes, gigabytes and floating_ips
      quotas returns general uninformative HTTP 500 error (LP: #1021373)
    - [7b215ed] Use default qemu-img cluster size in libvirt connection driver
    - [d3a87a2] Listing flavors with marker set returns 400 (LP: #956096)
    - [cf6a85a] nova-rootwrap hardcodes paths instead of using
      /sbin:/usr/sbin:/usr/bin:/bin (LP: #1013147)
    - [2efc87c] affinity filters don't work if scheduler_hints is None
      (LP: #1007573)
  ...

Read more...

Changed in nova (Ubuntu Precise):
status: Confirmed → Fix Released
Revision history for this message
Clint Byrum (clint-fewbar) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.