forget to release resource before rescheduling

Bug #1153964 reported by Tiantian Gao
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Tiantian Gao

Bug Description

The master code.

If an exception raised in libvirt._create_domain(), the resouce( like iptable rules, vif , etc.) will forever stay in the host.

An easy reproduction:
 0. setup all-in-one devstack.
 1. settting vncserver_listen = 1.2.3.4 in nova.conf. The configuration is to make libvirt._create_domain() raise an exception. In production environment, other reason can lead to an excetion. for example: (see attachment).

2. nova boot --flavor 1 --image XXXX vm1

3.
ubuntu@dev-stack:~$ nova list
+--------------------------------------+------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+------+--------+------------------+
| a8872a5a-4ba1-43fa-9ce3-6d52e7b166a1 | vm1 | ERROR | private=10.0.0.2 |
+--------------------------------------+------+--------+------------------+

nova show vm1
+-----------------------------+---------------------------------------------------------------------------------+
| Property | Value |
+-----------------------------+---------------------------------------------------------------------------------+
| status | ERROR |
| updated | 2013-03-12T08:31:20Z |
| OS-EXT-STS:task_state | None |
| key_name | None |
| image | cirros-0.3.1-x86_64-uec (f8b011a1-dac2-48be-ba8a-3d9241628bc1) |
| hostId | 409593d0d130d1be6902b6dd565e259e4c874ac03e1ba26c3a3c7787 |
| OS-EXT-STS:vm_state | error |
| flavor | m1.tiny (1) |
| id | a8872a5a-4ba1-43fa-9ce3-6d52e7b166a1 |
| security_groups | [{u'name': u'default'}] |
| user_id | 9694194fd660483ea427faf2da47795c |
| name | vm1 |
| created | 2013-03-12T08:31:04Z |
| fault | {u'message': u'NoValidHost', u'code': 500, u'created': u'2013-03-12T08:31:20Z'} |
| OS-DCF:diskConfig | MANUAL |
| metadata | {} |
| accessIPv4 | |
| accessIPv6 | |
| tenant_id | 6cc57820330644c98d606a15890678ff |
| OS-EXT-STS:power_state | 0 |
| OS-EXT-AZ:availability_zone | nova |
| config_drive | |
+-----------------------------+---------------------------------------------------------------------------------+

4. ubuntu@dev-stack:~$ sudo iptables-save |grep 10.0.0.2
-A nova-compute-local -d 10.0.0.2/32 -j nova-compute-inst-1

If there is more than one host, the instance will be rescheduled to other host. but the resources in this host will not release forever.

Revision history for this message
Tiantian Gao (gtt116) wrote :
Tiantian Gao (gtt116)
summary: - forget to release resource if exception in libvirt.spawn()
+ forget to release resource before rescheduling
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/24282

Changed in nova:
assignee: nobody → TianTian Gao (gtt116)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/24282
Committed: http://github.com/openstack/nova/commit/22e8e37e45c69136b252a494d8398b0b98e35b9e
Submitter: Jenkins
Branch: master

commit 22e8e37e45c69136b252a494d8398b0b98e35b9e
Author: gtt116 <gtt116@126.com>
Date: Wed Mar 13 03:43:16 2013 +0000

    Do cleaning up resource before rescheduling.

    Fix bug: #1153964

    If some exception raised when spawn instance, the instance will be
    rescheduled to other host. But just deallocate network before
    rescheduling is not enough, like security group filters and volumes
    will stay on old host.

    So shutdown instance and clean up volumes instead of just deallocate
    network before rescheduling.

    Change-Id: Ieb6220ae66f0b97a92c50f6998f456968b516fcb

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.