OpenStack Compute (nova)

shelve api does not work in the nova-cell environment

Bug #1338451 reported by Abhijeet Malawade
38
This bug affects 7 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Abhijeet Malawade
OpenStack Compute (nova) 2014.2 "juno"
Icehouse
Fix Released
Undecided
Abhijeet Malawade
OpenStack Compute (nova) 2014.1.3

Bug Description

If you run nova shelve api in nova-cell environment It throws following error:

Nova cell (n-cell-child) Logs:

2014-07-06 23:57:13.445 ERROR nova.cells.messaging [req-a689a1a1-4634-4634-974a-7343b5554f46 admin admin] Error processing message locally: save() got an unexpected keyword argument 'expected_task_state'
2014-07-06 23:57:13.445 TRACE nova.cells.messaging Traceback (most recent call last):
2014-07-06 23:57:13.445 TRACE nova.cells.messaging File "/opt/stack/nova/nova/cells/messaging.py", line 200, in _process_locally
2014-07-06 23:57:13.445 TRACE nova.cells.messaging resp_value = self.msg_runner._process_message_locally(self)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging File "/opt/stack/nova/nova/cells/messaging.py", line 1287, in _process_message_locally
2014-07-06 23:57:13.445 TRACE nova.cells.messaging return fn(message, **message.method_kwargs)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging File "/opt/stack/nova/nova/cells/messaging.py", line 700, in run_compute_api_method
2014-07-06 23:57:13.445 TRACE nova.cells.messaging return fn(message.ctxt, *args, **method_info['method_kwargs'])
2014-07-06 23:57:13.445 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 192, in wrapped
2014-07-06 23:57:13.445 TRACE nova.cells.messaging return func(self, context, target, *args, **kwargs)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 182, in inner
2014-07-06 23:57:13.445 TRACE nova.cells.messaging return function(self, context, instance, *args, **kwargs)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 163, in inner
2014-07-06 23:57:13.445 TRACE nova.cells.messaging return f(self, context, instance, *args, **kw)
2014-07-06 23:57:13.445 TRACE nova.cells.messaging File "/opt/stack/nova/nova/compute/api.py", line 2458, in shelve
2014-07-06 23:57:13.445 TRACE nova.cells.messaging instance.save(expected_task_state=[None])
2014-07-06 23:57:13.445 TRACE nova.cells.messaging TypeError: save() got an unexpected keyword argument 'expected_task_state'
2014-07-06 23:57:13.445 TRACE nova.cells.messaging

Nova compute log:

2014-07-07 00:05:19.084 ERROR oslo.messaging.rpc.dispatcher [req-9539189d-239b-4e74-8aea-8076740
31c2f admin admin] Exception during message handling: 'NoneType' object is not iterable
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _
dispatch_and_reply
    incoming.message))

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _
dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _
do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
    system_metadata, extra_usage_info)

  File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
    ignore_missing_network_data)

  File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
    macs = [vif['address'] for vif in nw_info]

TypeError: 'NoneType' object is not iterable

2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dis
t-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher payload)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 280, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher pass
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 266, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 330, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 308, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 296, in decorated_function
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 3847, in shelve_instance
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher self.conductor_api.notify_usage_exists(
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/conductor/api.py", line 206, in notify_usage_exists
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher system_metadata, extra_usage_info)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/conductor/rpcapi.py", line 320, in notify_usage_exists
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher extra_usage_info=extra_usage_info_p)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher retry=self.retry)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher timeout=timeout, retry=retry)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 401, in send
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher retry=retry)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 392, in _send
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher raise result
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher system_metadata, extra_usage_info)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher ignore_missing_network_data)
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher macs = [vif['address'] for vif in nw_info]
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher TypeError: 'NoneType' object is not iterable
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.084 TRACE oslo.messaging.rpc.dispatcher
2014-07-07 00:05:19.093 ERROR oslo.messaging._drivers.common [req-9539189d-239b-4e74-8aea-807674031c2f admin admin] Returning exception 'NoneType' object is not iterable
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
    incoming.message))

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/opt/stack/nova/nova/conductor/manager.py", line 351, in notify_usage_exists
    system_metadata, extra_usage_info)

  File "/opt/stack/nova/nova/compute/utils.py", line 250, in notify_usage_exists
    ignore_missing_network_data)

  File "/opt/stack/nova/nova/notifications.py", line 285, in bandwidth_usage
    macs = [vif['address'] for vif in nw_info]

TypeError: 'NoneType' object is not iterable
 to caller

Shelve api is failing in nova-cell environment because the compute_api shelve/unshelve
methods expect an Instance object, but cell is still passing the sqlalchemy form.

Also 'info_cache' and 'metadata' are not present in instance-object and shelve requires these
properties to be present in Instance object.

See original description
Abhijeet Malawade (abhijeet-malawade)
description: updated
Changed in nova:
assignee: nobody → Abhijeet Malawade (abhijeet-malawade)
tags: added: ntt
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/105112

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/105112
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=9d4b49c542e2076c8a572d1e8c6d50e255efe087
Submitter: Jenkins
Branch: master

commit 9d4b49c542e2076c8a572d1e8c6d50e255efe087
Author: Abhijeet Malawade <email address hidden>
Date: Wed Jul 2 08:03:31 2014 -0700

    shelve doesn't work on nova-cells environment

    The compute_api shelve/unshelve methods expect an Instance object,
    but cell is still passing the sqlalchemy form.
    So adding shelve/unshelve to the list of methods that will receive an
    Instance object when using nova-cell.

    When shelving instance with nova-cell environment,
    instance object needs to contain 'metadata' and 'info_cache'.
    So fetching the same from databse and adding it to
    instance object before calling shelve/unshelve methods.

    Closes-Bug: #1338451
    Change-Id: Ia31b33c9a1d347f15d3e0a67fa31ef6b25a7407a

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/icehouse)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/107612

Abhijeet Malawade (abhijeet-malawade)
tags: added: icehouse-backport-potential
Russell Bryant (russellb)
Changed in nova:
milestone: none → juno-2
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/icehouse)

Reviewed: https://review.openstack.org/107612
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e7d20879b2d4ef5261d0f85c520a63451b628eeb
Submitter: Jenkins
Branch: stable/icehouse

commit e7d20879b2d4ef5261d0f85c520a63451b628eeb
Author: Abhijeet Malawade <email address hidden>
Date: Wed Jul 2 08:03:31 2014 -0700

    shelve doesn't work on nova-cells environment

    The compute_api shelve/unshelve methods expect an Instance object,
    but cell is still passing the sqlalchemy form.
    So adding shelve/unshelve to the list of methods that will receive an
    Instance object when using nova-cell.

    When shelving instance with nova-cell environment,
    instance object needs to contain 'metadata' and 'info_cache'.
    So fetching the same from databse and adding it to
    instance object before calling shelve/unshelve methods.

    Conflicts:
        nova/cells/messaging.py

    Closes-Bug: #1338451
    Change-Id: Ia31b33c9a1d347f15d3e0a67fa31ef6b25a7407a
    (cherry picked from commit 9d4b49c542e2076c8a572d1e8c6d50e255efe087)

tags: added: in-stable-icehouse
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-2 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.