instance snapshot fails with "AttributeError: size" when using glance v1

[Matt Riedemann]

This is on a newton server deployed on 7/25 from master (newton).

I set use_glance_v1=True in nova.conf on the compute node.

I created a server from this image:

+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | ee1eca47dc88f4879d8a229cc70a07c6 |
| container_format | bare |
| created_at | 2016-07-06T17:25:08Z |
| disk_format | qcow2 |
| id | aac0314e-9bdf-4cee-a3d8-5f089008ea96 |
| locations | [{"url": "file:///var/lib/glance/images/aac0314e-9bdf-4cee-a3d8-5f089008ea96", |
| | "metadata": {}}] |
| min_disk | 0 |
| min_ram | 0 |
| name | cirros |
| owner | None |
| protected | False |
| size | 13287936 |
| status | active |
| tags | [] |
| updated_at | 2016-07-06T17:25:09Z |
| virtual_size | None |
| visibility | public |
+------------------+----------------------------------------------------------------------------------+

I tried to snapshot the instance and it failed with this:

http://paste.openstack.org/show/542180/

2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] Traceback (most recent call last):
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 231, in decorated_function
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] *args, **kwargs)
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 3024, in snapshot_instance
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] task_states.IMAGE_SNAPSHOT)
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/compute/manager.py", line 3054, in _snapshot_instance
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] update_task_state)
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1482, in snapshot
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] snapshot = self._image_api.get(context, image_id)
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/image/api.py", line 93, in get
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] show_deleted=show_deleted)
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/image/glance.py", line 266, in show
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] include_locations=include_locations)
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/image/glance.py", line 878, in _translate_from_glance
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] image, include_locations=include_locations)
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/nova/image/glance.py", line 968, in _extract_attributes
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] output[attr] = getattr(image, attr) or 0
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/glanceclient/openstack/common/apiclient/base.py", line 491, in __getattr__
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] return self.__getattr__(k)
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] File "/opt/bbc/openstack-11.0-master/nova/local/lib/python2.7/site-packages/glanceclient/openstack/common/apiclient/base.py", line 493, in __getattr__
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] raise AttributeError(k)
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd] AttributeError: size
2016-07-26 21:25:34.563 51383 ERROR nova.compute.manager [instance: 85bcc918-4d00-4d21-82fe-65b13035adcd]

Looking at that code, it looks like it assumes the 'size' attribute will be on the image, which it's not. Nova creates this image from the compute API:

https://github.com/openstack/nova/blob/9326c1ed403477d627fa1b94e6937c99deed9ecd/nova/compute/api.py#L2165

The extra_properties in this case come from the REST API, and I'm not passing any in in my snapshot request.

The other properties come from the instance's system_metadata, of which image_size isn't set:

mysql> select * from nova.instance_system_metadata where instance_uuid='85bcc918-4d00-4d21-82fe-65b13035adcd';
+---------------------+------------+------------+----+--------------------------------------+------------------------+--------------------------------------+---------+
| created_at | updated_at | deleted_at | id | instance_uuid | key | value | deleted |
+---------------------+------------+------------+----+--------------------------------------+------------------------+--------------------------------------+---------+
| 2016-07-26 20:54:08 | NULL | NULL | 92 | 85bcc918-4d00-4d21-82fe-65b13035adcd | image_disk_format | qcow2 | 0 |
| 2016-07-26 20:54:08 | NULL | NULL | 93 | 85bcc918-4d00-4d21-82fe-65b13035adcd | image_min_ram | 0 | 0 |
| 2016-07-26 20:54:08 | NULL | NULL | 94 | 85bcc918-4d00-4d21-82fe-65b13035adcd | image_min_disk | 10 | 0 |
| 2016-07-26 20:54:08 | NULL | NULL | 95 | 85bcc918-4d00-4d21-82fe-65b13035adcd | image_base_image_ref | aac0314e-9bdf-4cee-a3d8-5f089008ea96 | 0 |
| 2016-07-26 20:54:08 | NULL | NULL | 96 | 85bcc918-4d00-4d21-82fe-65b13035adcd | image_container_format | bare | 0 |
+---------------------+------------+------------+----+--------------------------------------+------------------------+--------------------------------------+---------+

So it looks like we shouldn't rely on image.size being set when using glance v1 and taking a snapshot.

[Matt Riedemann (mriedem)]

If you change this line:

https://github.com/openstack/nova/blob/9326c1ed403477d627fa1b94e6937c99deed9ecd/nova/image/glance.py#L968

To this:

 output[attr] = getattr(image, attr, 0) or 0

It works like a charm.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/347571

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/347571
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7a16c754fb49615b802e6b8f3d886847c168bd2a
Submitter: Jenkins
Branch: master

commit 7a16c754fb49615b802e6b8f3d886847c168bd2a
Author: Matt Riedemann <email address hidden>
Date: Tue Jul 26 18:27:48 2016 -0400

    Default image.size to 0 when extracting v1 image attributes

    When we snapshot a non-volume-backed instance, we create an
    image in nova.compute.api.API._create_image and set some
    values from the instance but 'size' isn't one of them.

    Later in the virt driver's snapshot method, at least for
    libvirt, it gets the snapshot image from the image API (glance)
    and when using glance v1 (use_glance_v1=True) the _extract_attributes
    method in nova.image.glance pulls the attributes out of the v1 image
    response to the form that nova expects. This code assumes that
    the 'size' attribute is set on the image, which for a snapshot image
    it might not be (yet anyway). This results in an AttributeError.

    This change defaults the size attribute value to 0 if it's not set.
    If it is set, but to None, we still use 0 as before.

    Change-Id: I14b0e44a7268231c2b19f013b563f0b8f09c2e88
    Closes-Bug: #1606707

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/359663

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 14.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/359663
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=056ab125c595aae42e60bb761a8324d9f245b1a4
Submitter: Jenkins
Branch: stable/mitaka

commit 056ab125c595aae42e60bb761a8324d9f245b1a4
Author: Matt Riedemann <email address hidden>
Date: Tue Jul 26 18:27:48 2016 -0400

    Default image.size to 0 when extracting v1 image attributes

    When we snapshot a non-volume-backed instance, we create an
    image in nova.compute.api.API._create_image and set some
    values from the instance but 'size' isn't one of them.

    Later in the virt driver's snapshot method, at least for
    libvirt, it gets the snapshot image from the image API (glance)
    and when using glance v1 (use_glance_v1=True) the _extract_attributes
    method in nova.image.glance pulls the attributes out of the v1 image
    response to the form that nova expects. This code assumes that
    the 'size' attribute is set on the image, which for a snapshot image
    it might not be (yet anyway). This results in an AttributeError.

    This change defaults the size attribute value to 0 if it's not set.
    If it is set, but to None, we still use 0 as before.

    Conflicts:
     nova/tests/unit/image/test_glance.py

    Original patch added test-case to TestExtractAttributes class,
    but stable/mitaka doesn't have that class.
    And that class used configurable value "use_glance_v1" that is
    not contained in stable/mitaka.
    So I added a test-case that tests fixed points to other existing
    class.

    Change-Id: I14b0e44a7268231c2b19f013b563f0b8f09c2e88
    Closes-Bug: #1606707
    (cherry picked from commit 7a16c754fb49615b802e6b8f3d886847c168bd2a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 13.1.2 release.