OpenStack Compute (nova)

Resource tracker: unable to start nova compute

Bug #1444439 reported by Gary Kotton on 2015-04-15

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Critical	Gary Kotton	OpenStack Compute (nova) 12.0.0 "liberty"
	Kilo	Fix Released	Critical	Unassigned	OpenStack Compute (nova) 2015.1.0 "kilo"

Bug Description

After a failure of the resize and a deletion of the instance. I am unable to restart the nova compute due to the exception below. The instance was deleted via nova api.

The DB is as follows:
mysql> select * from migrations;
+---------------------+---------------------+------------+----+------------------+------------------+---------------+----------------+--------------------------------------+----------------------+----------------------+------------------+------------------+---------+
| created_at | updated_at | deleted_at | id | source_compute | dest_compute | dest_host | status | instance_uuid | old_instance_type_id | new_instance_type_id | source_node | dest_node | deleted |
+---------------------+---------------------+------------+----+------------------+------------------+---------------+----------------+--------------------------------------+----------------------+----------------------+------------------+------------------+---------+
| 2015-04-15 09:44:02 | 2015-04-15 09:44:08 | NULL | 1 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | post-migrating | 42264e24-1385-41f1-8dfc-120a1891ab05 | 10 | 11 | domain-c167(DVS) | domain-c167(DVS) | 0 |
| 2015-04-15 09:48:13 | 2015-04-15 10:19:48 | NULL | 2 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | reverted | fcab4bde-d93e-4d79-ae35-9d1306da10a4 | 10 | 11 | domain-c167(DVS) | domain-c167(DVS) | 0 |
| 2015-04-15 10:23:56 | 2015-04-15 10:24:03 | NULL | 3 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | post-migrating | d074bbc0-b912-4c85-a02b-aabf56d45f0b | 10 | 11 | domain-c167(DVS) | domain-c167(DVS) | 0 |
| 2015-04-15 10:27:45 | 2015-04-15 10:28:16 | NULL | 4 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | reverted | 21e59c96-fa2f-45e3-9070-e982a2dafea6 | 10 | 11 | domain-c167(DVS) | domain-c167(DVS) | 0 |
| 2015-04-15 10:28:43 | 2015-04-15 10:29:16 | NULL | 5 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | confirming | 21e59c96-fa2f-45e3-9070-e982a2dafea6 | 10 | 11 | domain-c167(DVS) | domain-c167(DVS) | 0 |
| 2015-04-15 10:35:15 | 2015-04-15 10:53:16 | NULL | 6 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | confirmed | 4abd75b5-bb91-4ce7-a928-2a96941ea9cb | 10 | 14 | domain-c167(DVS) | domain-c167(DVS) | 0 |
| 2015-04-15 10:35:39 | 2015-04-15 10:53:17 | NULL | 7 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | confirmed | 5e01bddb-3978-4f6f-a4d3-6d24ed31afa4 | 14 | 10 | domain-c167(DVS) | domain-c167(DVS) | 0 |
| 2015-04-15 10:55:01 | 2015-04-15 10:55:02 | NULL | 8 | Ubuntu1404Server | Ubuntu1404Server | 10.160.94.173 | migrating | 20017567-5c83-4918-b269-525169009026 | 10 | 15 | domain-c167(DVS) | domain-c167(DVS) | 0 |
+---------------------+---------------------+------------+----+------------------+------------------+---------------+----------------+--------------------------------------+----------------------+----------------------+------------------+------------------+---------+
8 rows in set (0.00 sec)

2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 145, in wait
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup x.wait()
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 47, in wait
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/openstack/common/service.py", line 497, in run_service
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup service.start()
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/service.py", line 183, in start
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup self.manager.pre_start_hook()
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 1287, in pre_start_hook
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup self.update_available_resource(nova.context.get_admin_context())
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/manager.py", line 6236, in update_available_resource
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup rt.update_available_resource(context)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/resource_tracker.py", line 402, in update_available_resource
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup self._update_available_resource(context, resources)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup return f(*args, **kwargs)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/resource_tracker.py", line 445, in _update_available_resource
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup self._update_usage_from_migrations(context, resources, migrations)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/compute/resource_tracker.py", line 709, in _update_usage_from_migrations
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup instance = migration.instance
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/objects/migration.py", line 80, in instance
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup return objects.Instance.get_by_uuid(self._context, self.instance_uuid)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/objects/base.py", line 161, in wrapper
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup args, kwargs)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/conductor/rpcapi.py", line 325, in object_class_action
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup objver=objver, args=args, kwargs=kwargs)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup retry=self.retry)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup timeout=timeout, retry=retry)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup retry=retry)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup raise result
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup InstanceNotFound_Remote: Instance 42264e24-1385-41f1-8dfc-120a1891ab05 could not be found.
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/conductor/manager.py", line 423, in _object_dispatch
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup return getattr(target, method)(*args, **kwargs)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/objects/base.py", line 163, in wrapper
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup result = fn(cls, context, *args, **kwargs)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/objects/instance.py", line 564, in get_by_uuid
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup use_slave=use_slave)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/db/api.py", line 651, in instance_get_by_uuid
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup columns_to_join, use_slave=use_slave)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 233, in wrapper
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup return f(*args, **kwargs)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1744, in instance_get_by_uuid
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup columns_to_join=columns_to_join, use_slave=use_slave)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 1756, in _instance_get_by_uuid
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup raise exception.InstanceNotFound(instance_id=uuid)
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup InstanceNotFound: Instance 42264e24-1385-41f1-8dfc-120a1891ab05 could not be found.
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.821 TRACE nova.openstack.common.threadgroup
2015-04-15 04:47:04.824 INFO oslo_vmware.api [req-2d7c5d01-438c-494c-a382-b7c55b71d8be None None] Logging out and terminating the current session with ID = f4fcb.
nicira@Ubuntu1404Server:~/devstack$

Tags:

Gary Kotton (garyk) on 2015-04-15

Changed in nova:
importance:	Undecided → Critical
assignee:	nobody → Gary Kotton (garyk)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-15: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/173794

Changed in nova:
status:	New → In Progress

John Garbutt (johngarbutt) on 2015-04-15

tags:

added: kilo-rc-potential

Gary Kotton (garyk) on 2015-04-16

tags:

added: scheduler

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-20: Fix merged to nova (master)

Reviewed: https://review.openstack.org/173794
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ee7a7446cc6947a6bacacb6cb514934cc22e5782
Submitter: Jenkins
Branch: master

commit ee7a7446cc6947a6bacacb6cb514934cc22e5782
Author: Gary Kotton <email address hidden>
Date: Wed Apr 15 05:14:42 2015 -0700

Resource tracker: unable to restart nova compute

    The resource tracker calculates its used resources. In certain cases
    of failed migrations and an instance being deleted the resource tracker
    causes an exception in nova compute. If this situation arises then nova
    compute may not even be able to restart.

Change-Id: I4a154e0cae3b8e22bd59ed05ba708e07eed8dea7
Closes-bug: #1444439

Changed in nova:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-20: Fix proposed to nova (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/175360

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-22: Fix merged to nova (stable/kilo)

Reviewed: https://review.openstack.org/175360
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=389368bcfe498323b369f68682babb92a5b0ca54
Submitter: Jenkins
Branch: stable/kilo

commit 389368bcfe498323b369f68682babb92a5b0ca54
Author: Gary Kotton <email address hidden>
Date: Wed Apr 15 05:14:42 2015 -0700

Resource tracker: unable to restart nova compute

    Change-Id: I4a154e0cae3b8e22bd59ed05ba708e07eed8dea7
    Closes-bug: #1444439
    (cherry picked from commit ee7a7446cc6947a6bacacb6cb514934cc22e5782)

Thierry Carrez (ttx) on 2015-04-23

tags:

removed: kilo-rc-potential

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-04-30: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/179284

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-05-06: Fix merged to nova (master)

Download full text (18.1 KiB)

Reviewed: https://review.openstack.org/179284
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5228d4e418734164ffa5ccd91d2865d9cc659c00
Submitter: Jenkins
Branch: master

commit 906ab9d6522b3559b4ad36d40dec3af20397f223
Author: He Jie Xu <email address hidden>
Date: Thu Apr 16 07:09:34 2015 +0800

Update rpc version aliases for kilo

    Update all of the rpc client API classes to include a version alias
    for the latest version implemented in Kilo. This alias is needed when
    doing rolling upgrades from Kilo to Liberty. With this in place, you can
    ensure all services only send messages that both Kilo and Liberty will
    understand.

Closes-Bug: #1444745

Conflicts:
nova/conductor/rpcapi.py

NOTE(alex_xu): The conflict is due to there are some logs already added
into the master.

Change-Id: I2952aec9aae747639aa519af55fb5fa25b8f3ab4
(cherry picked from commit 78a8b5802ca148dcf37c5651f75f2126d261266e)

commit f191a2147a21c7e50926b288768a96900cf4c629
Author: Hans Lindgren <email address hidden>
Date: Fri Apr 24 13:10:39 2015 +0200

Add security group calls missing from latest compute rpc api version bump

The recent compute rpc api version bump missed out on the security group
related calls that are part of the api.

    One possible reason is that both compute and security group client side
    rpc api:s share a single target, which is of little value and only cause
    mistakes like this.

This change eliminates future problems like this by combining them into
one to get a 1:1 relationship between client and server api:s.

    Change-Id: I9207592a87fab862c04d210450cbac47af6a3fd7
    Closes-Bug: #1448075
    (cherry picked from commit bebd00b117c68097203adc2e56e972d74254fc59)

commit a2872a9262985bd0ee2c6df4f7593947e0516406
Author: Dan Smith <email address hidden>
Date: Wed Apr 22 09:02:03 2015 -0700

Fix migrate_flavor_data() to catch instances with no instance_extra rows

    The way the query was being performed previously, we would not see any
    instances that didn't have a row in instance_extra. This could happen if
    an instance hasn't been touched for several releases, or if the data
    set is old.

    The fix is a simple change to use outerjoin instead of join. This patch
    includes a test that ensures that instances with no instance_extra rows
    are included in the migration. If we query an instance without such a
    row, we create it before doing a save on the instance.

    Closes-Bug: #1447132
    Change-Id: I2620a8a4338f5c493350f26cdba3e41f3cb28de7
    (cherry picked from commit 92714accc49e85579f406de10ef8b3b510277037)

commit e3a7b83834d1ae2064094e9613df75e3b07d77cd
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Apr 23 02:18:41 2015 +0000

Updated from global requirements

Change-Id: I5d4acd36329fe2dccb5772fed3ec55b442597150

commit 8c9b5e620eef3233677b64cd234ed2551e6aa182
Author: Divya <email address hidden>
Date: Tue Apr 21 08:26:29 2015 +0200

Control create/delete flavor api permissions using policy.json

The permissions of ...

Reviewed:  https://review.openstack.org/179284
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5228d4e418734164ffa5ccd91d2865d9cc659c00
Submitter: Jenkins
Branch:    master

commit 906ab9d6522b3559b4ad36d40dec3af20397f223
Author: He Jie Xu <hejie.xu@intel.com>
Date:   Thu Apr 16 07:09:34 2015 +0800

Update rpc version aliases for kilo
    
    Update all of the rpc client API classes to include a version alias
    for the latest version implemented in Kilo.  This alias is needed when
    doing rolling upgrades from Kilo to Liberty.  With this in place, you can
    ensure all services only send messages that both Kilo and Liberty will
    understand.
    
    Closes-Bug: #1444745
    
    Conflicts:
    	nova/conductor/rpcapi.py
    
    NOTE(alex_xu): The conflict is due to there are some logs already added
    into the master.
    
    Change-Id: I2952aec9aae747639aa519af55fb5fa25b8f3ab4
    (cherry picked from commit 78a8b5802ca148dcf37c5651f75f2126d261266e)

commit f191a2147a21c7e50926b288768a96900cf4c629
Author: Hans Lindgren <hanlind@kth.se>
Date:   Fri Apr 24 13:10:39 2015 +0200

Add security group calls missing from latest compute rpc api version bump
    
    The recent compute rpc api version bump missed out on the security group
    related calls that are part of the api.
    
    One possible reason is that both compute and security group client side
    rpc api:s share a single target, which is of little value and only cause
    mistakes like this.
    
    This change eliminates future problems like this by combining them into
    one to get a 1:1 relationship between client and server api:s.
    
    Change-Id: I9207592a87fab862c04d210450cbac47af6a3fd7
    Closes-Bug: #1448075
    (cherry picked from commit bebd00b117c68097203adc2e56e972d74254fc59)

commit a2872a9262985bd0ee2c6df4f7593947e0516406
Author: Dan Smith <dansmith@redhat.com>
Date:   Wed Apr 22 09:02:03 2015 -0700

Fix migrate_flavor_data() to catch instances with no instance_extra rows
    
    The way the query was being performed previously, we would not see any
    instances that didn't have a row in instance_extra. This could happen if
    an instance hasn't been touched for several releases, or if the data
    set is old.
    
    The fix is a simple change to use outerjoin instead of join. This patch
    includes a test that ensures that instances with no instance_extra rows
    are included in the migration. If we query an instance without such a
    row, we create it before doing a save on the instance.
    
    Closes-Bug: #1447132
    Change-Id: I2620a8a4338f5c493350f26cdba3e41f3cb28de7
    (cherry picked from commit 92714accc49e85579f406de10ef8b3b510277037)

commit e3a7b83834d1ae2064094e9613df75e3b07d77cd
Author: OpenStack Proposal Bot <openstack-infra@lists.openstack.org>
Date:   Thu Apr 23 02:18:41 2015 +0000

Updated from global requirements
    
    Change-Id: I5d4acd36329fe2dccb5772fed3ec55b442597150

commit 8c9b5e620eef3233677b64cd234ed2551e6aa182
Author: Divya <dikonoor@in.ibm.com>
Date:   Tue Apr 21 08:26:29 2015 +0200

Control create/delete flavor api permissions using policy.json
    
    The permissions of create/delete flavor api is currently broken
    and expects the user to be always an admin, instead of controlling
    the permissions by the rules defined in the nova policy.json.
    
    Change-Id: Ide3c9ec2fa674b4fe3ea9d935cd4f7848914b82e
    Closes-Bug: 1445335
    (cherry picked from commit ced60b1d1b1608dc8229741b207a95498bc0b212)

commit bf79742d26ae66886bcdc55eeaf27e1d7ce24be5
Author: Przemyslaw Czesnowicz <przemyslaw.czesnowicz@intel.com>
Date:   Tue Apr 14 16:28:57 2015 +0100

Fix handling of pci_requests in consume_from_instance.
    
    Properly retrieve requests from pci_requests in consume_from_instance.
    Without this the call to numa_fit_instance_to_host will fail because
    it expects the request list.
    And change the order in which apply_requests and numa_fit_instance_to_host
    are called. Calling apply_requests first will remove devices from pools
    and  may make numa_fit_instance_to_host fail.
    
    Change-Id: I41cf4e8e5c1dea5f91e5261a8f5e88f46c7994ef
    Closes-bug: #1444021
    (cherry picked from commit 0913e799e9ce3138235f5ea6f80159f468ad2aaa)

commit c2d7060b480608d9773340f51d6496fadf97b667
Author: Przemyslaw Czesnowicz <przemyslaw.czesnowicz@intel.com>
Date:   Thu Apr 16 17:10:06 2015 +0100

Use list of requests in InstancePCIRequests.obj_from_db.
    
    InstancePCIRequests.obj_from_db assumes it's called with with a dict
    of values from instances_extra table, but in some cases it's called
    with just the value of pci_requests column.
    This changes obj_from_db to be used with just the value of pci_requests column.
    
    Change-Id: I7bed733c845c365081719a70b8a2f0cc9a58370c
    Closes-bug: #1445040
    (cherry picked from commit a074d7b4465b45730a5171e024c5c39a66a9c927)

commit 7a609f153808f7cee1edbbb36accc292fa8df0d0
Author: Przemyslaw Czesnowicz <przemyslaw.czesnowicz@intel.com>
Date:   Tue Apr 7 16:31:05 2015 +0100

Add numa_node field to PciDevicePool
    
    Without this field, PciDevicePool.from_dict will treat numa_node key in
    the dict as a tag, which in turn means that the scheduler client will
    drop it when converting stats to objects before reporting.
    
    Converting it back to dicts on the scheduler side thus will not have
    access to the numa_node information which would cause any requests that
    will look for the exact match between the device and instance NUMA nodes
    in the NUMATopologyFilter to fail.
    
    Closes-Bug: #1441169
    (cherry picked from commit 7db1ebc66c59205f78829d1e9cd10dcc1201d798)
    
    Conflicts:
    	nova/tests/unit/objects/test_objects.py
    
    Change-Id: I7381f909620e8e787178c0be9a362f8d3eb9ff7d

commit 880a356e40d327c0af4ce94b5a08fe0cd6fcab5d
Author: Nikola Dipanov <ndipanov@redhat.com>
Date:   Tue Apr 7 20:53:32 2015 +0100

scheduler: re-calculate NUMA on consume_from_instance
    
    This patch narrows down the race window between the filter running and
    the consumption of resources from the instance after the host has been
    chosen.
    
    It does so by re-calculating the fitted NUMA topology just before consuming it
    from the chosen host. Thus we avoid any locking, but also make sure that
    the host_state is kept as up to date as possible for concurrent
    requests, as there is no opportunity for switching threads inside a
    consume_from_instance.
    
    Several things worth noting:
      * Scheduler being lock free (and thus racy) does not really affect
      resources other than PCI and NUMA topology this badly - this is due
      to complexity of said resources. In order for scheduler decesions to not
      be based on basically guessing, in case of those two we will likely need
      to introduce either locking or special heuristics.
    
      * There is a lot of repeated code between the 'consume_from_instance'
      method and the actual filters. This situation should really be fixed but
      is out of scope for this bug fix (which is about preventing valid
      requests failing because of races in the scheduler).
    
    Change-Id: If0c7ad20506c9dddf4dec1eb64c9d6dd4fb75633
    Closes-bug: #1438238
    (cherry picked from commit d6b3156a6c89ddff9b149452df34c4b32c50b6c3)

commit a4e9a146c3993f5775501716a21632f34a63a3ad
Author: Rajesh Tailor <rajesh.tailor@nttdata.com>
Date:   Wed Apr 15 06:59:04 2015 -0700

Fix kwargs['migration'] KeyError in @errors_out_migration decorator
    
    @errors_out_migration decorator is used in the compute manager on
    resize_instance and finish_resize methods of ComputeManager class.
    It is decorated via @utils.expects_func_args('migration') to check
    'migration' is a parameter to the decorator method, however, that
    only ensures there is a migration argument, not that it's in args or
    kwargs (either is fine for what expects_func_args checks).
    The errors_out_migration decorator can get a KeyError when checking
    kwargs['migration'] and fails to set the migration status to 'error'.
    
    This fixes the KeyError in the decorator by normalizing the args/kwargs
    list into a single dict that we can pull the migration from.
    
    Change-Id: I774ac9b749b21085f4fbcafa4965a78d68eec9c7
    Closes-Bug: 1444300
    (cherry picked from commit 3add7923fc16c050d4cfaef98a87886c6b6a589c)

commit 389368bcfe498323b369f68682babb92a5b0ca54
Author: Gary Kotton <gkotton@vmware.com>
Date:   Wed Apr 15 05:14:42 2015 -0700

Resource tracker: unable to restart nova compute
    
    The resource tracker calculates its used resources. In certain cases
    of failed migrations and an instance being deleted the resource tracker
    causes an exception in nova compute. If this situation arises then nova
    compute may not even be able to restart.
    
    Change-Id: I4a154e0cae3b8e22bd59ed05ba708e07eed8dea7
    Closes-bug: #1444439
    (cherry picked from commit ee7a7446cc6947a6bacacb6cb514934cc22e5782)

commit bd6a40fecde943a3ded0124481a12c27dbb167de
Author: Andreas Jaeger <aj@suse.de>
Date:   Mon Apr 20 11:01:22 2015 +0200

Release Import of Translations from Transifex
    
    Manual import of Translations from Transifex. This change also removes
    all po files that are less than 66 per cent translated since such
    partially translated files will not help users.
    
    This updates also recreates all pot (translation source files) to
    reflect the state of the repository.
    
    This change needs to be done manually since the automatic import does
    not handle the proposed branches and we need to sync with latest
    translations.
    
    Change-Id: I0e9ef00182a2229602d23b8a67a02f0be62ee239

commit 8ebd515aa94ed399074a3b55bd36fd8cd579a499
Author: Matt Riedemann <mriedem@us.ibm.com>
Date:   Thu Apr 16 11:08:50 2015 -0700

Use kwargs from compute v4 proxy change_instance_metadata
    
    The args were passed to the compute manager method in the wrong order.
    We noticed this in the gate with KeyError: 'uuid' in the logs because of
    the LOG.debug statement in change_instance_metadata. Just use kwargs
    like rpcapi would normally.
    
    There isn't a unit test for this since the v4 proxy code goes away in
    liberty, this is for getting it into stable/kilo.
    
    Closes-Bug: #1444728
    
    Change-Id: Ic988f48d99e626ee5773c97904e09dbf00c5414a
    (cherry picked from commit e55f746ea8590cce7c2b07a023197f369251a7ef)

commit b19764d2c6a8160102a806c1d6811c4182a8bac8
Author: Matt Riedemann <mriedem@us.ibm.com>
Date:   Wed Apr 15 11:51:26 2015 -0700

compute: stop handling virt lifecycle events in cleanup_host()
    
    When rebooting a compute host, guest VMs can be getting shutdown
    automatically by the hypervisor and the virt driver is sending events to
    the compute manager to handle them. If the compute service is still up
    while this happens it will try to call the stop API to power off the
    instance and update the database to show the instance as stopped.
    
    When the compute service comes back up and events come in from the virt
    driver that the guest VMs are running, nova will see that the vm_state
    on the instance in the nova database is STOPPED and shut down the
    instance by calling the stop API (basically ignoring what the virt
    driver / hypervisor tells nova is the state of the guest VM).
    
    Alternatively, if the compute service shuts down after changing the
    intance task_state to 'powering-off' but before the stop API cast is
    complete, the instance can be in a strange vm_state/task_state
    combination that requires the admin to manually reset the task_state to
    recover the instance.
    
    Let's just try to avoid some of this mess by disconnecting the event
    handling when the compute service is shutting down like we do for
    neutron VIF plugging events. There could still be races here if the
    compute service is shutting down after the hypervisor (e.g. libvirtd),
    but this is at least a best attempt to do the mitigate the potential
    damage.
    
    Closes-Bug: #1444630
    Related-Bug: #1293480
    Related-Bug: #1408176
    
    Change-Id: I1a321371dff7933cdd11d31d9f9c2a2f850fd8d9
    (cherry picked from commit d1fb8d0fbdd6cb95c43b02f754409f1c728e8cd0)

commit 75e9de5d572578520c217b540aa2a40726f137f0
Author: Roman Podoliaka <rpodolyaka@mirantis.com>
Date:   Wed Mar 4 17:27:06 2015 +0200

Forbid booting of QCOW2 images with virtual_size > root_gb
    
    Currently, it's possible to boot an instance from a QCOW2 image,
    which has virtual_size bigger than one allowed by the given flavor
    (root_gb).
    
    The issue is caused by two different problems in the code:
    
    1) typo in get_disk_size() has made it always return None and
       effectively disabled verify_base_size() checks
    
    2) Rbd image backend skips the verify_base_size() step for
       'cached' images (the one with base files), so it is possible to
       boot an instance using a larger flavor once and then use smaller
       flavors to boot the same image, even if allowed root_gb size is
       smaller than the image virtual size
    
    Closes-Bug: #1429093
    
    Change-Id: I383130e5f8cc288f4b428ed43fe4d3aba7169473
    (cherry picked from commit c1f9ed27af64e6893d9d0153a964df5aba99b8f0)

commit 33ba90240a2ad3165274d9e54ceb156273404c9a
Author: Matt Riedemann <mriedem@us.ibm.com>
Date:   Mon Apr 13 14:47:20 2015 -0700

Pass migrate_data to pre_live_migration
    
    Commit ebfa09fa197a1d88d1b3ab1f308232c3df7dc009 added an RPC proxy but
    as part of that was passing migrate_data=None for pre_live_migration
    which breaks live block migration when not using shared storage.
    
    Closes-Bug: #984996
    
    Change-Id: I2a83f1fb0e4468f9a6c67a188af725c3406139d1
    (cherry picked from commit 4e515ec2269a1c3187ee9ffad3a6be059ec74b0b)

commit dea6116723f22632c2e478e00bb0aafcd2febdc9
Author: Timofey Durakov <tdurakov@mirantis.com>
Date:   Fri Apr 10 19:38:33 2015 +0300

Fixed order of arguments during execution live_migrate()
    
    order of arguments that passed to
    ComputeManager.live_,migration() differs in ComputeManager and
    _ComputeV4Proxy classes
    
    Change-Id: I23c25d219e9cdd0673ae6a12250219680fb7bda9
    Closes-Bug:#1442656
    (cherry picked from commit ba521fa53711774e0718808fe333aca676de57ae)

commit 22d7547c6b62fb9dabd861e4941edd34eedabfc6
Author: Doug Hellmann <doug@doughellmann.com>
Date:   Wed Apr 15 19:58:17 2015 +0000

update .gitreview for stable/kilo
    
    Change-Id: I6356513ac42b79402dbde8ee5e75cbbd1aee7eef

commit 68d6f924037f3b931add2ce5d0d433913e720ca6
Author: Ken'ichi Ohmichi <oomichi@mxs.nes.nec.co.jp>
Date:   Wed Apr 15 03:13:43 2015 +0000

Add min/max of API microversions to version API
    
    As nova-spec api-microversions, versions API needs to expose minimum
    and maximum microversions to version API, because clients need to
    know available microversions through the API. That is very important
    for the interoperability.
    This patch adds these versions as the nova-spec mentioned.
    
    Note:
      As v2(not v2.1) API change manner, we have added new extensions if
      changing API. However, this patch doesn't add a new extension even
      if adding new parameters "version" and "min_version" because version
      API is independent from both v2 and v2.1 APIs.
    
    Change-Id: Id464a07d624d0e228fe0aa66a04c8e51f292ba0c
    Closes-Bug: #1443375
    (cherry picked from commit 1830870718fe7472b47037f3331cfe59b5bdda07)
    (cherry picked from commit 853671e912c6ad9a4605acad2575417911875cdd)

commit d82d492c67db546d01addc5fead9708760fb6abd
Author: Sabari Kumar Murugesan <smurugesan@vmware.com>
Date:   Mon Apr 6 20:36:48 2015 -0700

VMware: Fix attribute error in resize
    
    The class DatastorePath was recently removed from ds_util as it's
    available in oslo.vmware. One of the reference was missed during
    the refactor.
    
    Change-Id: Idc5825c304a99e83cbf36e93751148d6f995131a
    Closes-Bug: #1440968
    (cherry picked from commit ab4a5a5300179a79f7a67688f0e9f3fc280c0efa)

commit 3cff2c673c6cdf487c2a1eb2a5c6c89c6de80d11
Author: jichenjc <jichenjc@cn.ibm.com>
Date:   Fri Mar 20 08:36:37 2015 +0800

Release bdm constraint source and dest type
    
    https://bugs.launchpad.net/nova/+bug/1377958 fixed a problem
    that source_type: image, destination_type: local is not
    supported for boot instance, exception should be raised to
    reject the param otherwise it will lead to instance become
    ERROR state.
    
    However the fix introduced a problem on nova client
    https://bugs.launchpad.net/python-novaclient/+bug/1418484
    The fix of the bug leads to following command become invalid
    
    nova boot test-vm --flavor m1.medium --image centos-vm-32
    --nic net-id=c3f40e33-d535-4217-916b-1450b8cd3987 --block-device
    id=26b7b917-2794-452a-95e5-2efb2ca6e32d,bus=sata,source=volume,bootindex=1
    
    So we need to release the original constraint to allow
    the above special case pass the validation check then
    we can revert the nova client exception
    (https://review.openstack.org/#/c/165932/)
    
    This patch checks the boot_index and whether image param is
    given after we found the bdm has source_type: image,
    destination_type: local, if this is the special case, then
    no exception will be raised.
    
    Closes-Bug: #1433609
    
    Change-Id: If43faae95169bc3864449a8364975f5c887aac14
    (cherry picked from commit cadbcc440a2fcfb8532f38111999a06557fbafc2)

commit 97145ba175291d6522afa079860c72220e43024a
Author: Dan Smith <dansmith@redhat.com>
Date:   Fri Apr 10 07:10:52 2015 -0700

Fix check_can_live_migrate_destination() in ComputeV4Proxy
    
    There was a mismatch in the V4 proxy in the call signatures of this
    function. This was missed because the "destination" parameter is passed
    in the rpcapi as the host to contact, which is consumed by the rpc
    layer and not passed. Since it was not called one of the standard
    names (either host if to be not passed, or host_param if to be passed),
    this was missed.
    
    Change-Id: Idf2160934dade650ed02b672f3b64cb26247f8e6
    Closes-Bug: #1442602
    (cherry picked from commit 0c08f7f2ef070f7c6172d7742f9789e0a8bda91a)

Thierry Carrez (ttx) on 2015-06-24