OpenStack Compute (Nova)

Failure while rebuilding an instance with libvirt

Reported by Stanislaw Pitucha on 2011-10-04
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Naveed Massjouni
Diablo
Undecided
Unassigned

Bug Description

On the Diablo release, trying to run 'nova rebuild' on an instance running on libvirt results in a failure with:

2011-10-04 16:57:15,543 DEBUG nova.virt.libvirt_conn [-] instance instance-00000001: starting toXML method from (pid=18743) to_xml /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:1162
2011-10-04 16:57:15,543 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 597, in spawn
(nova.exception): TRACE: block_device_info=block_device_info)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 1164, in to_xml
(nova.exception): TRACE: block_device_info)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 1058, in _prepare_xml_info
(nova.exception): TRACE: block_device_info)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/driver.py", line 58, in block_device_info_get_mapping
(nova.exception): TRACE: block_device_mapping = block_device_info.get('block_device_mapping') or []
(nova.exception): TRACE: AttributeError: 'tuple' object has no attribute 'get'
(nova.exception): TRACE:
2011-10-04 16:57:15,575 DEBUG nova.rpc [-] Making asynchronous cast on notifications.error... from (pid=18743) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:747
[...]
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/notifier/api.py", line 122, in notify
(nova.exception): TRACE: driver.notify(msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/notifier/rabbit_notifier.py", line 36, in notify
(nova.exception): TRACE: rpc.cast(context, topic, message)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 49, in cast
(nova.exception): TRACE: return get_impl().cast(context, topic, msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 750, in cast
(nova.exception): TRACE: conn.topic_send(topic, msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 465, in topic_send
(nova.exception): TRACE: self.publisher_send(TopicPublisher, topic, msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 433, in publisher_send
(nova.exception): TRACE: publisher.send(msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 234, in send
(nova.exception): TRACE: self.producer.publish(msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 121, in publish
(nova.exception): TRACE: compression, headers)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 144, in _prepare
(nova.exception): TRACE: body) = encode(body, serializer=serializer)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/serialization.py", line 119, in encode
(nova.exception): TRACE: payload = encoder(data)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/anyjson/__init__.py", line 123, in <lambda>
(nova.exception): TRACE: serialize = lambda value: implementation.serialize(value)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/anyjson/__init__.py", line 85, in serialize
(nova.exception): TRACE: raise TypeError(*exc.args)
(nova.exception): TRACE: TypeError: <capsule object "virConnectPtr" at 0x434fe70> is not JSON serializable
(nova.exception): TRACE:
2011-10-04 16:57:15,584 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.rpc): TRACE: return f(*args, **kw)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 574, in rebuild_instance
(nova.rpc): TRACE: self.driver.spawn(context, instance_ref, network_info, bd_mapping)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 129, in wrapped
(nova.rpc): TRACE: raise Error(str(e))
(nova.rpc): TRACE: Error: 'tuple' object has no attribute 'get'
(nova.rpc): TRACE:

Ahmad Hassan (ahmad-hassan-q) wrote :

I fixed that one. I made a change in rebuild_instance method and fixed the bug. I will the changes for review.

Thanks.

--Ahmad
HP Cloud services

Thierry Carrez (ttx) on 2011-10-21
Changed in nova:
assignee: nobody → Ahmad Hassan (ahmad-hassan-q)
importance: Undecided → High
status: New → Confirmed
David Kranz (david-kranz) wrote :

One of our users just reported this. It seems the same as bug 904724, which is currently marked as invalid. Has this actually been fixed anywhere? It seems like it would be a candidate for diablo stable.

Ahmad Hassan (ahmad-hassan-q) wrote :

Hi David,

We have fixed it in our local repositories but I think we haven't pushed it upstream yet.

--Ahmad

Mark McLoughlin (markmc) wrote :

Apparently this is already fixed in essex. See https://review.openstack.org/2414

I'm not sure I buy that, though

https://github.com/openstack/nova/blob/a5e73ffe/nova/compute/manager.py#L724

        bd_mapping = self._setup_block_device_mapping(context, instance)
        ....
        self.driver.spawn(context, instance, image_meta,
                          network_info, bd_mapping)

https://github.com/openstack/nova/blob/a5e73ffe/nova/compute/manager.py#L275

    def _setup_block_device_mapping(self, context, instance):
        ...
        return (swap, ephemerals, block_device_mapping)

https://github.com/openstack/nova/blob/a5e73ffe/nova/virt/libvirt/connection.py#L670

    def spawn(self, context, instance, image_meta, network_info,
              block_device_info=None):
        xml = self.to_xml(instance, network_info, image_meta, False,
                          block_device_info=block_device_info)

https://github.com/openstack/nova/blob/a5e73ffe/nova/virt/libvirt/connection.py#L1201

    def to_xml(self, instance, network_info, image_meta=None, rescue=False,
               block_device_info=None):
        ...
        xml_info = self._prepare_xml_info(instance, network_info, image_meta,
                                          rescue, block_device_info)

https://github.com/openstack/nova/blob/a5e73ffe/nova/virt/libvirt/connection.py#L1087

    def _prepare_xml_info(self, instance, network_info, image_meta, rescue,
                          block_device_info=None):
        block_device_mapping = driver.block_device_info_get_mapping(
            block_device_info)

https://github.com/openstack/nova/blob/a5e73ffe/nova/virt/driver.py#L56

    def block_device_info_get_mapping(block_device_info):
        block_device_info = block_device_info or {}
        block_device_mapping = block_device_info.get('block_device_mapping') or []
        return block_device_mapping

Looks obvious to me that the bug is still there. But https://review.openstack.org/2414 didn't fix the issue either

Thierry Carrez (ttx) wrote :

@markmc: care to repropose it for master ?

Changed in nova:
assignee: Ahmad Hassan (ahmad-hassan-q) → Naveed Massjouni (ironcamel)

Fix proposed to branch: master
Review: https://review.openstack.org/3420

Changed in nova:
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/3420
Committed: http://github.com/openstack/nova/commit/f7346b6ce8a945499d9f2f3eb0d531f9da0705dc
Submitter: Jenkins
Branch: master

commit f7346b6ce8a945499d9f2f3eb0d531f9da0705dc
Author: Naveed Massjouni <email address hidden>
Date: Wed Jan 25 18:48:58 2012 +0000

    Fixing rebuilds on libvirt.

    bug: 867627
    Change-Id: I663fd4004a2198c15d576de33b5aa35188a6e74a

Changed in nova:
status: In Progress → Fix Committed

Fix proposed to branch: master
Review: https://review.openstack.org/3427

Changed in nova:
status: Fix Committed → In Progress

Reviewed: https://review.openstack.org/3427
Committed: http://github.com/openstack/nova/commit/a27dd1b24ca67de28cced3d8d2eec98f86cbb958
Submitter: Jenkins
Branch: master

commit a27dd1b24ca67de28cced3d8d2eec98f86cbb958
Author: Naveed Massjouni <email address hidden>
Date: Wed Jan 25 20:15:28 2012 +0000

    Fixing rebuilds on libvirt, seriously.

    bug: 867627
    Change-Id: I497b18a56997e347995ad8869855127bd31cb974

Changed in nova:
status: In Progress → Fix Committed
Vish Ishaya (vishvananda) wrote :

looks like this just needs a backport

summary: - Failure while rebuilding an instance with livirt
+ Failure while rebuilding an instance with libvirt
tags: added: low-hanging-fruit
Thierry Carrez (ttx) on 2012-02-29
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers