OpenStack Compute (Nova)

Setting non blocking libvirt flag causes __str__ error on virConnect

Reported by Devin Carlen on 2012-03-23
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Yun Mao
Essex
High
Unassigned

Bug Description

If you set the --libvirt_nonblocking flag to true and attempt to launch an instance, the following exception is thrown:

(nova.rpc.amqp): TRACE: AttributeError: virConnect instance has no attribute '__str__'
(nova.rpc.amqp): TRACE: File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 219, in __str__
(nova.rpc.amqp): TRACE: return self._obj.__str__()

yong sheng gong (gongysh) wrote :

with trunk codes I failed to reproduce it on redhat 6.2:
[root@robinlinux ~]# rpm -qa | grep libvirt
libvirt-client-0.9.6-2.el6.x86_64
libvirt-java-devel-0.4.7-1.el6.noarch
libvirt-python-0.9.6-2.el6.x86_64
libvirt-0.9.6-2.el6.x86_64
libvirt-java-0.4.7-1.el6.noarch
libvirt-devel-0.9.6-2.el6.x86_64
[root@robinlinux ~]# python --version
Python 2.6.6

and devstack:
libvirt 0.9.2, python 2.7.2+

my nova.conf is:
[DEFAULT]
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lib/nova/tmp
instances_path=/var/lib/nova/instances
debug=True
scheduler_driver=nova.scheduler.simple.SimpleScheduler
dhcpbridge_flagfile=/etc/nova/nova.conf
fixed_range=10.10.10.0/24
s3_host=127.0.0.1
osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions
my_ip=127.0.0.1
public_interface=br100
vlan_interface=eth0
sql_connection=mysql://nova:nova@localhost/nova
instance_name_template=instance-%08x
novncproxy_base_url=http://127.0.0.1:6080/vnc_auto.html
xvpvncproxy_base_url=http://127.0.0.1:6081/console
vncserver_listen=127.0.0.1
vncserver_proxyclient_address=127.0.0.1
api_paste_config=/etc/nova/api-paste.ini
image_service=nova.image.glance.GlanceImageService
ec2_dmz_host=127.0.0.1
rabbit_host=localhost
rabbit_password=guest
glance_api_servers=127.0.0.1:9292
connection_type=libvirt
libvirt_type=kvm
libvirt_nonblocking=true
.....

Thierry Carrez (ttx) wrote :

@Devin: Could you give a bit more detail on the environment you reproduce the issue on ?

Changed in nova:
importance: Critical → High
status: New → Incomplete
yong sheng gong (gongysh) wrote :
Download full text (4.0 KiB)

Haha, I reproduce it today.

My way is not to start nova-network, and then run nova boot command.

below is message:
2012-03-28 08:12:22 ERROR nova.compute.manager [req-6274f7dc-4815-4926-b913-b16cba0cc4d6 44c3d88f257a4dc39790c318fcb1a2cc 36cf36c3972a437890e5df7bf9805097] [instance: a26c20ed-72b1-45bc-a7c5-b321824b50b1] Instance failed network setup
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/compute/manager.py", line 559, in _allocate_network
(nova.compute.manager): TRACE: requested_networks=requested_networks)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/network/api.py", line 170, in allocate_for_instance
(nova.compute.manager): TRACE: 'args': args})
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/__init__.py", line 68, in call
(nova.compute.manager): TRACE: return _get_impl().call(context, topic, msg, timeout)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/impl_kombu.py", line 674, in call
(nova.compute.manager): TRACE: return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/amqp.py", line 338, in call
(nova.compute.manager): TRACE: rv = list(rv)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/amqp.py", line 299, in __iter__
(nova.compute.manager): TRACE: self._iterator.next()
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/impl_kombu.py", line 572, in iterconsume
(nova.compute.manager): TRACE: yield self.ensure(_error_callback, _consume)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/impl_kombu.py", line 503, in ensure
(nova.compute.manager): TRACE: error_callback(e)
(nova.compute.manager): TRACE: File "/root/test/git/nova/nova/rpc/impl_kombu.py", line 553, in _error_callback
(nova.compute.manager): TRACE: raise rpc_common.Timeout()
(nova.compute.manager): TRACE: Timeout: Timeout while waiting on RPC response.
(nova.compute.manager): TRACE:
2012-03-28 08:12:22 ERROR nova.rpc.amqp [req-6274f7dc-4815-4926-b913-b16cba0cc4d6 44c3d88f257a4dc39790c318fcb1a2cc 36cf36c3972a437890e5df7bf9805097] Exception during message handling
(nova.rpc.amqp): TRACE: Traceback (most recent call last):
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/rpc/amqp.py", line 252, in _process_data
(nova.rpc.amqp): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/exception.py", line 138, in wrapped
(nova.rpc.amqp): TRACE: payload)
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/notifier/api.py", line 118, in notify
(nova.rpc.amqp): TRACE: payload = utils.to_primitive(payload, convert_instances=True)
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/utils.py", line 798, in to_primitive
(nova.rpc.amqp): TRACE: level=level)
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/utils.py", line 792, in to_primitive
(nova.rpc.amqp): TRACE: level=level))
(nova.rpc.amqp): TRACE: File "/root/test/git/nova/nova/utils.py", line 813, in to_primitive
(nova.r...

Read more...

Thierry Carrez (ttx) on 2012-03-29
Changed in nova:
status: Incomplete → Confirmed
Yun Mao (yunmao) wrote :

This is due to libvirt using old-style class while the proxy_call method in eventlet tpool assumes all objects are in new style.

Changed in nova:
assignee: nobody → Yun Mao (yunmao)
Thierry Carrez (ttx) on 2012-04-03
tags: removed: essex-rc-potential

Fix proposed to branch: master
Review: https://review.openstack.org/6164

Changed in nova:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/6164
Committed: http://github.com/openstack/nova/commit/bc173ec44e6c3a42af22976122b46fd25546559b
Submitter: Jenkins
Branch: master

commit bc173ec44e6c3a42af22976122b46fd25546559b
Author: Yun Mao <email address hidden>
Date: Tue Apr 3 11:17:50 2012 -0400

    Fixed bug 962840, added a test case.

    eventlet.tpool.Proxy doesn't work with old-style class in __str__()
    or __repr__() calls. See bug #962840 for details.
    We perform a monkey patch to replace those two instance methods.

    Change-Id: Ia51bbd3e71cad7df45da5b3b27eef70f9d9e9002

Changed in nova:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/6309
Committed: http://github.com/openstack/nova/commit/3b14c746910abf95e8136e409085873fd6ac970d
Submitter: Jenkins
Branch: stable/essex

commit 3b14c746910abf95e8136e409085873fd6ac970d
Author: Yun Mao <email address hidden>
Date: Tue Apr 3 11:17:50 2012 -0400

    Fixed bug 962840, added a test case.

    eventlet.tpool.Proxy doesn't work with old-style class in __str__()
    or __repr__() calls. See bug #962840 for details.
    We perform a monkey patch to replace those two instance methods.

    Change-Id: Ia51bbd3e71cad7df45da5b3b27eef70f9d9e9002

tags: added: in-stable-essex
Thierry Carrez (ttx) on 2012-05-23
Changed in nova:
milestone: none → folsom-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-09-27
Changed in nova:
milestone: folsom-1 → 2012.2
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers