auto confirm is invalid with nova master branch

Bug #1202975 reported by wangpan
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
wangpan

Bug Description

Reproduce steps in my devstack env with latest master branch nova:
1. set resize_confirm_window=10 in nova.conf
2. create an instance
3. resize it
4. wait for it is confirmed and watch the log

2013-07-19 08:34:50.775 INFO nova.compute.manager [-] Found 1 unconfirmed migrations older than 10 seconds
2013-07-19 08:34:50.776 INFO nova.compute.manager [-] Automatically confirming migration 21 for instance d85c25a4-c4a7-40c3-99f2-750834ed332a
......
2013-07-19 08:34:50.863 ERROR nova.compute.manager [-] [instance: d85c25a4-c4a7-40c3-99f2-750834ed332a] Error auto-confirming resize: u'\'system_metadata\'\nTraceback (most recent call last):\n\n File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 426, in _process_data\n **args)
File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)\n\n File "/opt/stack/nova/nova/conductor/manager.py", line 522, in compute_confirm_resize
    self.compute_api.confirm_resize(context, instance, migration_ref)
  File "/opt/stack/nova/nova/compute/api.py", line 188, in wrapped
    return func(self, context, target, *args, **kwargs)\n\n File "/opt/stack/nova/nova/compute/api.py", line 178, in inner
    return function(self, context, instance, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/api.py", line 159, in inner
    return f(self, context, instance, *args, **kw)
  File "/opt/stack/nova/nova/compute/api.py", line 2032, in confirm_resize
  File "/opt/stack/nova/nova/compute/api.py", line 2108, in _downsize_quota_delta
    \'new_\')
  File "/opt/stack/nova/nova/compute/flavors.py", line 242, in extract_flavor
    sys_meta = utils.instance_sys_meta(instance)
  File "/opt/stack/nova/nova/utils.py", line 1048, in instance_sys_meta
   if isinstance(instance[\'system_metadata\'], dict):
KeyError: \'system_metadata\'\n'. Will retry later.

Revision history for this message
wangpan (hzwangpan) wrote :

The last commit of the codes I can reproduce this bug is:
commit 5ada7de0c74ab44d15ae9b18a9b521662b382d5b
Merge: 7c28952 7e8fa8f
Author: Jenkins <email address hidden>
Date: Fri Jul 19 01:21:35 2013 +0000

    Merge "Fix filtering aggregate metadata by key"

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/37862

Changed in nova:
assignee: nobody → wangpan (hzwangpan)
status: New → In Progress
Matt Riedemann (mriedem)
tags: added: compute unified-objects
Revision history for this message
wangpan (hzwangpan) wrote :

2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 2437, in confirm_resize
2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp "resize.confirm.start")
2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 1403, in _notify_about_instance_usage
2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp extra_usage_info=extra_usage_info, host=self.host)
2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/utils.py", line 248, in notify_about_instance_usage
2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp network_info, system_metadata, **extra_usage_info)
2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/notifications.py", line 383, in info_from_instance
2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp instance_info['metadata'] = instance_ref['metadata']
2013-08-20 02:27:13.125 TRACE nova.openstack.common.rpc.amqp KeyError: 'metadata'

Revision history for this message
wangpan (hzwangpan) wrote :

the backtrace above is the reason why the 'metadata' attr should be added, too.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/37862
Committed: http://github.com/openstack/nova/commit/38e267264a888dfddc2414b6dcc55e53d086b4b0
Submitter: Jenkins
Branch: master

commit 38e267264a888dfddc2414b6dcc55e53d086b4b0
Author: Wangpan <email address hidden>
Date: Fri Jul 19 17:49:21 2013 +0800

    Fixes auto confirm invalid error

    In the confirm_resize method at the compute API level, the 'metadata' attribute
    is used to notify the instance usage and the 'system_metadata' attribute is
    needed to get the old/new instance type values and the old vm_state, but the
    auto confirm periodic task doesn't get the instance object with these two
    attributes. This results in 'KeyError' exception being raised when the auto
    confirm periodic task runs.
    This commit sets expected_attrs to get 'metadata' and 'system_metadata' when
    retrieving the instance object in the _poll_unconfirmed_resizes compute manager
    method.

    Reproduce steps in my devstack env with latest master branch nova:
    1. set resize_confirm_window=10 in nova.conf
    2. create an instance
    3. resize it
    4. wait for it is confirmed and watch the log

    Fixes bug 1202975

    Change-Id: I940f195d45fba4f2d25f9c517f11e99b74e97a8f

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → havana-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: havana-3 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.