Failure while rebuilding an instance with libvirt

Bug #867627 reported by Stanislaw Pitucha
22
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Naveed Massjouni

Bug Description

On the Diablo release, trying to run 'nova rebuild' on an instance running on libvirt results in a failure with:

2011-10-04 16:57:15,543 DEBUG nova.virt.libvirt_conn [-] instance instance-00000001: starting toXML method from (pid=18743) to_xml /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:1162
2011-10-04 16:57:15,543 ERROR nova.exception [-] Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 597, in spawn
(nova.exception): TRACE: block_device_info=block_device_info)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 1164, in to_xml
(nova.exception): TRACE: block_device_info)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py", line 1058, in _prepare_xml_info
(nova.exception): TRACE: block_device_info)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/virt/driver.py", line 58, in block_device_info_get_mapping
(nova.exception): TRACE: block_device_mapping = block_device_info.get('block_device_mapping') or []
(nova.exception): TRACE: AttributeError: 'tuple' object has no attribute 'get'
(nova.exception): TRACE:
2011-10-04 16:57:15,575 DEBUG nova.rpc [-] Making asynchronous cast on notifications.error... from (pid=18743) cast /usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py:747
[...]
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/notifier/api.py", line 122, in notify
(nova.exception): TRACE: driver.notify(msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/notifier/rabbit_notifier.py", line 36, in notify
(nova.exception): TRACE: rpc.cast(context, topic, message)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 49, in cast
(nova.exception): TRACE: return get_impl().cast(context, topic, msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 750, in cast
(nova.exception): TRACE: conn.topic_send(topic, msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 465, in topic_send
(nova.exception): TRACE: self.publisher_send(TopicPublisher, topic, msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 433, in publisher_send
(nova.exception): TRACE: publisher.send(msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 234, in send
(nova.exception): TRACE: self.producer.publish(msg)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 121, in publish
(nova.exception): TRACE: compression, headers)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/messaging.py", line 144, in _prepare
(nova.exception): TRACE: body) = encode(body, serializer=serializer)
(nova.exception): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/serialization.py", line 119, in encode
(nova.exception): TRACE: payload = encoder(data)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/anyjson/__init__.py", line 123, in <lambda>
(nova.exception): TRACE: serialize = lambda value: implementation.serialize(value)
(nova.exception): TRACE: File "/usr/lib/pymodules/python2.7/anyjson/__init__.py", line 85, in serialize
(nova.exception): TRACE: raise TypeError(*exc.args)
(nova.exception): TRACE: TypeError: <capsule object "virConnectPtr" at 0x434fe70> is not JSON serializable
(nova.exception): TRACE:
2011-10-04 16:57:15,584 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 620, in _process_data
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 98, in wrapped
(nova.rpc): TRACE: return f(*args, **kw)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 117, in decorated_function
(nova.rpc): TRACE: function(self, context, instance_id, *args, **kwargs)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 574, in rebuild_instance
(nova.rpc): TRACE: self.driver.spawn(context, instance_ref, network_info, bd_mapping)
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 129, in wrapped
(nova.rpc): TRACE: raise Error(str(e))
(nova.rpc): TRACE: Error: 'tuple' object has no attribute 'get'
(nova.rpc): TRACE:

Revision history for this message
Ahmad Hassan (ahmad-hassan-q) wrote :

I fixed that one. I made a change in rebuild_instance method and fixed the bug. I will the changes for review.

Thanks.

--Ahmad
HP Cloud services

Thierry Carrez (ttx)
Changed in nova:
assignee: nobody → Ahmad Hassan (ahmad-hassan-q)
importance: Undecided → High
status: New → Confirmed
Revision history for this message
David Kranz (david-kranz) wrote :

One of our users just reported this. It seems the same as bug 904724, which is currently marked as invalid. Has this actually been fixed anywhere? It seems like it would be a candidate for diablo stable.

Revision history for this message
Ahmad Hassan (ahmad-hassan-q) wrote :

Hi David,

We have fixed it in our local repositories but I think we haven't pushed it upstream yet.

--Ahmad

Revision history for this message
Mark McLoughlin (markmc) wrote :

Apparently this is already fixed in essex. See https://review.openstack.org/2414

I'm not sure I buy that, though

https://github.com/openstack/nova/blob/a5e73ffe/nova/compute/manager.py#L724

        bd_mapping = self._setup_block_device_mapping(context, instance)
        ....
        self.driver.spawn(context, instance, image_meta,
                          network_info, bd_mapping)

https://github.com/openstack/nova/blob/a5e73ffe/nova/compute/manager.py#L275

    def _setup_block_device_mapping(self, context, instance):
        ...
        return (swap, ephemerals, block_device_mapping)

https://github.com/openstack/nova/blob/a5e73ffe/nova/virt/libvirt/connection.py#L670

    def spawn(self, context, instance, image_meta, network_info,
              block_device_info=None):
        xml = self.to_xml(instance, network_info, image_meta, False,
                          block_device_info=block_device_info)

https://github.com/openstack/nova/blob/a5e73ffe/nova/virt/libvirt/connection.py#L1201

    def to_xml(self, instance, network_info, image_meta=None, rescue=False,
               block_device_info=None):
        ...
        xml_info = self._prepare_xml_info(instance, network_info, image_meta,
                                          rescue, block_device_info)

https://github.com/openstack/nova/blob/a5e73ffe/nova/virt/libvirt/connection.py#L1087

    def _prepare_xml_info(self, instance, network_info, image_meta, rescue,
                          block_device_info=None):
        block_device_mapping = driver.block_device_info_get_mapping(
            block_device_info)

https://github.com/openstack/nova/blob/a5e73ffe/nova/virt/driver.py#L56

    def block_device_info_get_mapping(block_device_info):
        block_device_info = block_device_info or {}
        block_device_mapping = block_device_info.get('block_device_mapping') or []
        return block_device_mapping

Looks obvious to me that the bug is still there. But https://review.openstack.org/2414 didn't fix the issue either

Revision history for this message
Mark McLoughlin (markmc) wrote :
Revision history for this message
Thierry Carrez (ttx) wrote :

@markmc: care to repropose it for master ?

Changed in nova:
assignee: Ahmad Hassan (ahmad-hassan-q) → Naveed Massjouni (ironcamel)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/3420

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/3420
Committed: http://github.com/openstack/nova/commit/f7346b6ce8a945499d9f2f3eb0d531f9da0705dc
Submitter: Jenkins
Branch: master

commit f7346b6ce8a945499d9f2f3eb0d531f9da0705dc
Author: Naveed Massjouni <email address hidden>
Date: Wed Jan 25 18:48:58 2012 +0000

    Fixing rebuilds on libvirt.

    bug: 867627
    Change-Id: I663fd4004a2198c15d576de33b5aa35188a6e74a

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/3427

Changed in nova:
status: Fix Committed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/3427
Committed: http://github.com/openstack/nova/commit/a27dd1b24ca67de28cced3d8d2eec98f86cbb958
Submitter: Jenkins
Branch: master

commit a27dd1b24ca67de28cced3d8d2eec98f86cbb958
Author: Naveed Massjouni <email address hidden>
Date: Wed Jan 25 20:15:28 2012 +0000

    Fixing rebuilds on libvirt, seriously.

    bug: 867627
    Change-Id: I497b18a56997e347995ad8869855127bd31cb974

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Vish Ishaya (vishvananda) wrote :

looks like this just needs a backport

summary: - Failure while rebuilding an instance with livirt
+ Failure while rebuilding an instance with libvirt
tags: added: low-hanging-fruit
Thierry Carrez (ttx)
Changed in nova:
milestone: none → essex-4
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: essex-4 → 2012.1
Sean Dague (sdague)
no longer affects: nova/diablo
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.