Evacuate libvirt instance failed with error 'Cannot load 'disk_format' in the base class'

[Christine Wang]

openstack-nova-12.0.0-201509202117

When evacuate a libvirt instance, it failed with the following error: NotImplementedError: Cannot load 'disk_format' in the base class

2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2431, in spawn
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher block_device_info)
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 630, in get_disk_info
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher rescue)
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 537, in get_disk_mapping
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher disk_bus, cdrom_bus, root_device_name)
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/blockinfo.py", line 432, in get_root_info
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher if image_meta.disk_format == 'iso':
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 66, in getter
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher self.obj_load_attr(name)
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 555, in obj_load_attr
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher _("Cannot load '%s' in the base class") % attrname)
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher NotImplementedError: Cannot load 'disk_format' in the base class
2015-09-30 08:04:47.484 19026 ERROR oslo_messaging.rpc.dispatcher

When libvirt instance is evacuated, the image_meta is passed in with {}. So, the disk_format is not populated with the ImageMeta object.

It's unclear to me what's the right way to fix this issue. Should change ImageMeta's from_dict to make sure 'disk_format' is always populated or we should add obj_load_attr method to ImageMeta

[Mark Doffman (mjdoffma)]

I believe this has already been fixed. See change https://review.openstack.org/#/c/188119/ which has already been merged in to master and will be available in the Liberty release.

[Christine Wang (ijuwang)]

I do have that fix. That's why I hit this issue.

The image_mata got passed to spawn is {}.

So, when image_meta = objects.ImageMeta.from_dict(image_meta), it doesn't populate the disk_format attribute.

So, in the get_root_info of the stack, it does this ==>
 if image_meta.disk_format == 'iso':
So, I got the NotImplementedError: Cannot load 'disk_format' in the base class error

[Matt Riedemann (mriedem)]

(2:26:41 PM) mriedem: but where does image_meta get turned into an object?
(2:26:56 PM) ijuwang: spawn
(2:26:58 PM) ijuwang: in the driver
(2:27:26 PM) bauwser: yey
(2:27:50 PM) ijuwang: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2427
(2:27:52 PM) bauwser: mriedem: https://review.openstack.org/#/c/188119/
(2:27:58 PM) mriedem: https://github.com/openstack/nova/blob/stable/liberty/nova/virt/libvirt/driver.py#L2427
(2:28:01 PM) mriedem: yeah
(2:29:33 PM) bauwser: mriedem: so, you see, it seems we have a regression...
(2:30:13 PM) mriedem: so....
(2:30:25 PM) bauwser: Critical I guess
(2:30:40 PM) bauwser: hence me hassling folks like mriedem
(2:30:43 PM) mriedem: image_meta (the dict) must not have disk_format in it, or it's None and ImageMeta.from_dict defaults it to {}
(2:30:51 PM) mriedem: which basically creates an empty ImageMeta object
(2:30:59 PM) mriedem: so you'd have to lazy-load any attributes on it
(2:31:56 PM) mriedem: b/c image_ref is coming in as None from the API, yeah, so image_meta is {} when we convert to an object
(2:32:05 PM) bauwser: right
(2:32:07 PM) mriedem: ok

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/230625

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/230657

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230625
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0717d064a50ae489a2db75d665c811e9e12c37f7
Submitter: Jenkins
Branch: master

commit 0717d064a50ae489a2db75d665c811e9e12c37f7
Author: Matt Riedemann <email address hidden>
Date: Fri Oct 2 13:00:43 2015 -0700

    libvirt: check if ImageMeta.disk_format is set before accessing it

    Commit 4f43bbafa4bfc2ab149d5db4eb55320c1602658b changed
    blockinfo.get_root_info to access the disk_format attribute on the
    passed in image_meta (ImageMeta object) like an object, but if that
    attribute is not set you'll get an error.

    image_meta can be an empty object in the case of evacuate where
    image_ref is passed as None to the compute manager's rebuild_instance
    method in the case of evacuate, so image_meta defaults to {}. So when
    the ImageMeta object is created in LibvirtDriver.spawn, there is nothing
    in it.

    We have to check if the attribute is set before accessing it using
    obj_attr_is_set because the ImageMeta class does not include the
    NovaObjectDictCompat mixin, so we can't use
    image_meta.get('disk_format').

    Closes-Bug: #1501831

    Change-Id: I95e543415b881603bb997ae0e159017c03e58a53

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/230657
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c24bf9e469dbc2df8e8b8900015197dc7c388e8c
Submitter: Jenkins
Branch: stable/liberty

commit c24bf9e469dbc2df8e8b8900015197dc7c388e8c
Author: Matt Riedemann <email address hidden>
Date: Fri Oct 2 13:00:43 2015 -0700

    libvirt: check if ImageMeta.disk_format is set before accessing it

    Commit 4f43bbafa4bfc2ab149d5db4eb55320c1602658b changed
    blockinfo.get_root_info to access the disk_format attribute on the
    passed in image_meta (ImageMeta object) like an object, but if that
    attribute is not set you'll get an error.

    image_meta can be an empty object in the case of evacuate where
    image_ref is passed as None to the compute manager's rebuild_instance
    method in the case of evacuate, so image_meta defaults to {}. So when
    the ImageMeta object is created in LibvirtDriver.spawn, there is nothing
    in it.

    We have to check if the attribute is set before accessing it using
    obj_attr_is_set because the ImageMeta class does not include the
    NovaObjectDictCompat mixin, so we can't use
    image_meta.get('disk_format').

    Closes-Bug: #1501831

    Change-Id: I95e543415b881603bb997ae0e159017c03e58a53
    (cherry picked from commit 2b1a2bc034e371a9f6d95f48ed8fa77e0854d675)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/235181

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/235181
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=d8a3b9d6408627ce9a293d1e62d003757af655a1
Submitter: Jenkins
Branch: master

commit 6df6ad3ff32f2b1fe2978df1032002548ad8eb66
Author: Davanum Srinivas <email address hidden>
Date: Wed Oct 7 08:11:35 2015 -0700

    Omnibus stable/liberty fix

    There are currently 3 different blocking issues in stable/liberty due
    to library releases: webob 1.5, oslo.db 3.0.0, and
    oslo.versionedobjects 0.11.0. This is a squashed fix for all of them
    as none can land without the others.

    Issue #1 - oslo.db

    Add testresources used by oslo.db fixture

    If we use oslo.db fixtures, we'll need the package or
    the next version of oslo.db release will break us.

    (Cherry-picked from 4bcc26487837b7ece7797f88622dea1b6d09bd94)

    Closes-Bug: #1503501

    Issue #2 - oslo.versionedobjects

    Drop unused obj_to_primitive() override

    This was a band-aid override until o.vo gained the obj_relationships fix
    that this method overrides. That has been in place since o.vo 0.8.0, which
    means this is long since no longer necessary (and is actually blocking our
    ability to absorb bug fixes to this code in o.vo). Further, we no longer
    use this directly because we're doing backports based on version manifests,
    which means we no longer consult child_versions _or_ obj_relationships.

    (cherry picked from commit 142f1d9cc4ace90956c665c40b1f78795f9f7e29)

    Issue #3 - webob

    Default ConvertedException code to 500

    webob 1.5.0 released on 10/11 has change f6c749011 which
    strictly enforces status codes in exceptions, and 0 is not
    a valid status code so tests fail.

    Change the default to 500 to match the default in the parent
    class in webob.

    Closes-Bug: #1505153
    (cherry picked from commit 10438c0fc34bd088e018e1a5e8ec57b396528792)

    Change-Id: I1e06e77308a7dd23209124f0807d61fb52470188

commit 606204354b5ed96852240020769c81acda9f9fc8
Author: Matt Riedemann <email address hidden>
Date: Mon Oct 5 20:32:58 2015 +0000

    Revert "[libvirt] Move cleanup of imported files to imagebackend"

    This reverts commit 9ba70756de326ffaa8be43acfde12cad04ed0af2

    The change introduced an UnboundLocalError if we fail to
    create the config_drive_image variable. Also, the original
    change didn't have any unit tests and came late in the
    liberty release so I don't really want to mess with fixing
    this given we need the fix in liberty-rc2.

    Change-Id: Ia7b70aa139b67cf58b5c0f9fbcd2a4deb465914e
    Closes-Bug: #1502961

commit ef655379445693443146f8a3ed31cabb011d9937
Author: OpenStack Proposal Bot <email address hidden>
Date: Thu Oct 8 06:41:06 2015 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: Idcac653033ab9808e06451a0dd690db4736834b2

commit eda3029aa74932f421d2992ac24f5ac3c92f347c
Author: Dan Smith <email address hidden>
Date: Tue Oct 6 10:58:18 2...

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.