OpenStack Compute (nova)

PCI passthrough failing on extra_info

Bug #1223559 reported by David Kang
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Yongli He
OpenStack Compute (nova) 2013.2 "havana"

Bug Description

Havana3 is installed using Packstack on CentOS 6.4.
Nova-compute dies right after start with error "NameError: global name '_' is not defined".

Here is the info:

* /etc/nova/nova.conf:
pci_alias={"name":"test", "product_id":"7190", "vendor_id":"8086", "device_type":"ACCEL"}

pci_passthrough_whitelist=[{"vendor_id":"8086","product_id":"7190"}]

 With that configuration, nova-compute fails with the following log:

* /var/log/nova/compute.log:

  File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
    **args)

  File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
    result = getattr(proxyobj, method)(ctxt, **kwargs)

  File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 567, in object_action
    result = getattr(objinst, objmethod)(context, *args, **kwargs)

  File "/usr/lib/python2.6/site-packages/nova/objects/base.py", line 141, in wrapper
    return fn(self, ctxt, *args, **kwargs)

  File "/usr/lib/python2.6/site-packages/nova/objects/pci_device.py", line 242, in save
    self._from_db_object(context, self, db_pci)

NameError: global name '_' is not defined
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup x.wait()
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 166, in wait
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 192, in main
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 65, in run_service
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup service.start()
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/service.py", line 164, in start
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup self.manager.pre_start_hook()
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 805, in pre_start_hook
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup self.update_available_resource(nova.context.get_admin_context())
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 4773, in update_available_resource
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup rt.update_available_resource(context)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 246, in inner
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup return f(*args, **kwargs)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py", line 318, in update_available_resource
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup self._sync_compute_node(context, resources)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py", line 347, in _sync_compute_node
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup self._update(context, resources, prune_stats=True)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/compute/resource_tracker.py", line 420, in _update
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup self.pci_tracker.save(context)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/pci/pci_manager.py", line 126, in save
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup dev.save(context)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/objects/base.py", line 134, in wrapper
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup ctxt, self, fn.__name__, args, kwargs)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/conductor/rpcapi.py", line 497, in object_action
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup objmethod=objmethod, args=args, kwargs=kwargs)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/rpcclient.py", line 85, in call
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup return self._invoke(self.proxy.call, ctxt, method, **kwargs)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/rpcclient.py", line 63, in _invoke
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup return cast_or_call(ctxt, msg, **self.kwargs)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/proxy.py", line 126, in call
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup result = rpc.call(context, real_topic, msg, timeout)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 139, in call
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup return _get_impl().call(CONF, context, topic, msg, timeout)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 794, in call
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup rpc_amqp.get_connection_pool(conf, Connection))
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 574, in call
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup rv = list(rv)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 539, in __iter__
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup raise result
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup NameError: global name '_' is not defined
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup **args)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 567, in object_action
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup result = getattr(objinst, objmethod)(context, *args, **kwargs)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/objects/base.py", line 141, in wrapper
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup return fn(self, ctxt, *args, **kwargs)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.6/site-packages/nova/objects/pci_device.py", line 242, in save
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup self._from_db_object(context, self, db_pci)
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup NameError: global name '_' is not defined

Yongli He (yongli-he)
Changed in nova:
assignee: nobody → Yongli He (yongli-he)
Revision history for this message
Yongli He (yongli-he) wrote :

hi, David
could you paste the new trace to the bug ? (note it with the patch) that's close to the fix i think.

diff --git a/nova/objects/pci_device.py b/nova/objects/pci_device.py
index a83b8f3..d0a628a 100644
--- a/nova/objects/pci_device.py
+++ b/nova/objects/pci_device.py
@@ -145,7 +145,7 @@ class PciDevice(base.NovaPersistentObject,
base.NovaObject):
             if key != 'extra_info':
                 pci_device[key] = db_dev[key]
             else:
- extra_info = db_dev.get("extra_info")
+ extra_info = db_dev.get("extra_info", '{}')
                 pci_device.extra_info = jsonutils.loads(extra_info)
         pci_device._context = context
         pci_device.obj_reset_changes()

--
Russell Bryant

Revision history for this message
Russell Bryant (russellb) wrote :

might need to be this instead ...

- extra_info = db_dev.get("extra_info")
+ extra_info = db_dev.get("extra_info") or '{}'

tags: added: unified-objects
Russell Bryant (russellb)
summary: - pci passthrough fails with "NameError: global name '_' is not defined"
+ PCI passthrough failing on extra_info
Revision history for this message
David Kang (dkang) wrote : Re: [Bug 1223559] Re: pci passthrough fails with "NameError: global name '_' is not defined"
Download full text (19.7 KiB)

----- Original Message -----
> From: "Russell Bryant" <email address hidden>
> To: <email address hidden>
> Sent: Wednesday, September 11, 2013 6:23:29 AM
> Subject: [Bug 1223559] Re: pci passthrough fails with "NameError: global name '_' is not defined"
> might need to be this instead ...
>
> - extra_info = db_dev.get("extra_info")
> + extra_info = db_dev.get("extra_info") or '{}'
>
>
> ** Tags added: unified-objects
>
> --
> You received this bug notification because you are subscribed to the
> bug
> report.
> https://bugs.launchpad.net/bugs/1223559
>
> Title:
> pci passthrough fails with "NameError: global name '_' is not defined"
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
>
> Havana3 is installed using Packstack on CentOS 6.4.
> Nova-compute dies right after start with error "NameError: global name
> '_' is not defined".
>
> Here is the info:
>
> * /etc/nova/nova.conf:
> pci_alias={"name":"test", "product_id":"7190", "vendor_id":"8086",
> "device_type":"ACCEL"}
>
> pci_passthrough_whitelist=[{"vendor_id":"8086","product_id":"7190"}]
>
>
> With that configuration, nova-compute fails with the following log:
>
> * /var/log/nova/compute.log:
>
> File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py",
> line 461, in _process_data
> **args)
>
> File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py",
> line 172, in dispatch
> result = getattr(proxyobj, method)(ctxt, **kwargs)
>
> File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py",
> line 567, in object_action
> result = getattr(objinst, objmethod)(context, *args, **kwargs)
>
> File "/usr/lib/python2.6/site-packages/nova/objects/base.py", line
> 141, in wrapper
> return fn(self, ctxt, *args, **kwargs)
>
> File "/usr/lib/python2.6/site-packages/nova/objects/pci_device.py",
> line 242, in save
> self._from_db_object(context, self, db_pci)
>
> NameError: global name '_' is not defined
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
> Traceback (most recent call last):
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
> File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py",
> line 117, in wait
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
> x.wait()
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
> File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py",
> line 49, in wait
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
> return self.thread.wait()
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
> File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line
> 166, in wait
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
> return self._exit_event.wait()
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
> File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116,
> in wait
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
> return hubs.get_hub().switch()
> 2013-09-10 12:52:23.774 14749 TRACE nova.openstack.common.threadgroup
>...

Revision history for this message
David Kang (dkang) wrote : Re: [Bug 1223559] Re: pci passthrough fails with "NameError: global name '_' is not defined"
Download full text (33.4 KiB)

> From: "David Kang" <email address hidden>
> To: <email address hidden>
> Sent: Wednesday, September 11, 2013 10:07:52 AM
> Subject: Re: [Bug 1223559] Re: pci passthrough fails with "NameError: global name '_' is not defined"
> ----- Original Message -----
> > From: "Russell Bryant" <email address hidden>
> > To: <email address hidden>
> > Sent: Wednesday, September 11, 2013 6:23:29 AM
> > Subject: [Bug 1223559] Re: pci passthrough fails with "NameError:
> > global name '_' is not defined"
> > might need to be this instead ...
> >
> > - extra_info = db_dev.get("extra_info")
> > + extra_info = db_dev.get("extra_info") or '{}'
> >
> >
> > ** Tags added: unified-objects
> >
> > --
> > You received this bug notification because you are subscribed to the
> > bug
> > report.
> > https://bugs.launchpad.net/bugs/1223559
> >
> > Title:
> > pci passthrough fails with "NameError: global name '_' is not
> > defined"
> >
> > Status in OpenStack Compute (Nova):
> > New
> >
> > Bug description:
> >
> > Havana3 is installed using Packstack on CentOS 6.4.
> > Nova-compute dies right after start with error "NameError: global
> > name
> > '_' is not defined".
> >
> > Here is the info:
> >
> > * /etc/nova/nova.conf:
> > pci_alias={"name":"test", "product_id":"7190", "vendor_id":"8086",
> > "device_type":"ACCEL"}
> >
> > pci_passthrough_whitelist=[{"vendor_id":"8086","product_id":"7190"}]
> >
> >
> > With that configuration, nova-compute fails with the following log:
> >
> > * /var/log/nova/compute.log:
> >
> > File
> > "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py",
> > line 461, in _process_data
> > **args)
> >
> > File
> > "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py",
> > line 172, in dispatch
> > result = getattr(proxyobj, method)(ctxt, **kwargs)
> >
> > File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py",
> > line 567, in object_action
> > result = getattr(objinst, objmethod)(context, *args, **kwargs)
> >
> > File "/usr/lib/python2.6/site-packages/nova/objects/base.py", line
> > 141, in wrapper
> > return fn(self, ctxt, *args, **kwargs)
> >
> > File "/usr/lib/python2.6/site-packages/nova/objects/pci_device.py",
> > line 242, in save
> > self._from_db_object(context, self, db_pci)
> >
> > NameError: global name '_' is not defined
> > 2013-09-10 12:52:23.774 14749 TRACE
> > nova.openstack.common.threadgroup
> > Traceback (most recent call last):
> > 2013-09-10 12:52:23.774 14749 TRACE
> > nova.openstack.common.threadgroup
> > File
> > "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py",
> > line 117, in wait
> > 2013-09-10 12:52:23.774 14749 TRACE
> > nova.openstack.common.threadgroup
> > x.wait()
> > 2013-09-10 12:52:23.774 14749 TRACE
> > nova.openstack.common.threadgroup
> > File
> > "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py",
> > line 49, in wait
> > 2013-09-10 12:52:23.774 14749 TRACE
> > nova.openstack.common.threadgroup
> > return self.thread.wait()
> > 2013-09-10 12:52:23.774 14749 TRACE
> > nova.openstack.common.threadgroup
> > File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py",
> > line
> > 166, in wait
> > 2013-09-10 12:52:23.77...

Revision history for this message
jiang, yunhong (yunhong-jiang) wrote :

Yongli, we need below patch, I'm OOP today and have no time to fully test it. A quick check seems it works.

diff --git a/nova/objects/pci_device.py b/nova/objects/pci_device.py
index a83b8f3..b3ae328 100644
--- a/nova/objects/pci_device.py
+++ b/nova/objects/pci_device.py
@@ -136,8 +136,8 @@ class PciDevice(base.NovaPersistentObject, base.NovaObject):

     def __init__(self):
         super(PciDevice, self).__init__()
- self.extra_info = {}
         self.obj_reset_changes()
+ self.extra_info = {}

     @staticmethod
     def _from_db_object(context, pci_device, db_dev):

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/46464

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/46464
Committed: http://github.com/openstack/nova/commit/0663778c531a69bf55ed8f88cd828febd69be44c
Submitter: Jenkins
Branch: master

commit 0663778c531a69bf55ed8f88cd828febd69be44c
Author: He Yongli <email address hidden>
Date: Fri Sep 13 17:35:59 2013 +0800

    pci passthrough fails while trying to decode extra_info

    pci device with empty extra_info should use null dict instead of
    None. keep the init of extra_info to make db could access it.

    Closes-Bug: #1223559

    Change-Id: I2e57eb53dfd4f5168d52dc94c2057fc575458aee
    Signed-off-by: Yongli He <email address hidden>

Changed in nova:
status: In Progress → Fix Committed
Russell Bryant (russellb)
Changed in nova:
milestone: none → havana-rc1
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: havana-rc1 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.