Schema Missing kernel_id, ramdisk_id causes #1447193

Bug #1447215 reported by Jorge Niedbalski on 2015-04-22
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Glance
High
Jorge Niedbalski
Kilo
Critical
Flavio Percoco
glance (Ubuntu)
Undecided
Unassigned
Vivid
Undecided
Jorge Niedbalski

Bug Description

[Description]

[Environment]

- Ubuntu 14.04.2
- OpenStack Kilo

ii glance 1:2015.1~rc1-0ubuntu2~cloud0 all OpenStack Image Registry and Delivery Service - Daemons
ii glance-api 1:2015.1~rc1-0ubuntu2~cloud0 all OpenStack Image Registry and Delivery Service - API
ii glance-common 1:2015.1~rc1-0ubuntu2~cloud0 all OpenStack Image Registry and Delivery Service - Common
ii glance-registry 1:2015.1~rc1-0ubuntu2~cloud0 all OpenStack Image Registry and Delivery Service - Registry
ii python-glance 1:2015.1~rc1-0ubuntu2~cloud0 all OpenStack Image Registry and Delivery Service - Python library
ii python-glance-store 0.4.0-0ubuntu1~cloud0 all OpenStack Image Service store library - Python 2.x
ii python-glanceclient 1:0.15.0-0ubuntu1~cloud0 all Client library for Openstack glance server.

[Steps to reproduce]

0) Set /etc/glance/glance-api.conf to enable_v2_api=False
1) nova boot --flavor m1.small --image base-image --key-name keypair --availability-zone nova --security-groups default snapshot-bug
2
) nova image-create snapshot-bug snapshot-bug-instance

At this point the created image has no kernel_id (None) and image_id (None)

3) Enable_v2_api=True in glance-api.conf and restart.

4) Run a os-image-api=2 client,

$ glance --os-image-api-version 2 image-list

This will fail with #1447193

[Description]

The schema-image.json file needs to be modified to allow null, string values for both attributes.

Related branches

Changed in glance:
assignee: nobody → Jorge Niedbalski (niedbalski)

Fix proposed to branch: master
Review: https://review.openstack.org/176379

Changed in glance:
status: New → In Progress
Samuel Matzek (smatzek) wrote :

I used slightly different reproduction steps.

# glance --os-image-api-version 1 image-list
+----+------+-------------+------------------+------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+----+------+-------------+------------------+------+--------+
+----+------+-------------+------------------+------+--------+

# glance --os-image-api-version 1 image-create --name test --disk-format raw --container-format bare --property ramdisk_id= --property kernel_id= --file ./junkfile
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| Property 'kernel_id' | None |
| Property 'ramdisk_id' | None |
| checksum | 0632d04c8ec91b3c8c4ea5ae1e38908a |
| container_format | bare |
| created_at | 2015-06-05T20:03:54.817722 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | 1438711d-2e4d-47f3-a092-d7da541c2451 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | test |
| owner | 963177847ec149c9b84f488140c737cd |
| protected | False |
| size | 5 |
| status | active |
| updated_at | 2015-06-05T20:03:55.411470 |
| virtual_size | None |
+-----------------------+--------------------------------------+

# glance --os-image-api-version 2 image-list
None is not of type u'string'

Failed validating u'type' in schema[u'properties'][u'ramdisk_id']:
    {u'description': u'ID of image stored in Glance that should be used as the ramdisk when booting an AMI-style image.',
     u'is_base': False,
     u'pattern': u'^([0-9a-fA-F]){8}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){4}-([0-9a-fA-F]){12}$',
     u'type': u'string'}

On instance[u'ramdisk_id']:
    None

The proposed fix https://review.openstack.org/#/c/176379/2 fixes the problem.

All VM snapshot images created by libvirt KVM driver in Nova that did not have ramdisk or kernel_id values will exhibit this behavior and they will fail to boot/spawn/deploy when Nova moves to use Glance v2.

Image using the rbd ephemeral snapshot code, https://review.openstack.org/#/c/125963/, are already exhibiting the failure behavior because this spawn path uses a Glance v2 show call to get image locations.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in glance (Ubuntu):
status: New → Confirmed
Samuel Matzek (smatzek) wrote :

I'm going to mention this bug, as I did on #1447193 that I'm using a new glanceclient:
# glance --version
0.17.1

While I'm adding yet another comment I thought I'd drop in the nova stack for how this fails spawn in code that uses clone and image locations.

  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2388, in spawn
    admin_pass=admin_password)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2786, in _create_image
    instance, size, fallback_from_host)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5916, in _try_fetch_image_cache
    size=size)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 231, in cache
    *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 737, in create_image
    prepare_template(target=base, max_size=size, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
    fetch_func(target=target, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2778, in clone_fallback_to_fetch
    backend.clone(context, disk_images['image_id'])
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 762, in clone
    include_locations=True)
  File "/usr/lib/python2.7/site-packages/nova/image/api.py", line 93, in get
    show_deleted=show_deleted)
  File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 301, in show
    _reraise_translated_image_exception(image_id)
  File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 299, in show
    image = self._client.call(context, version, 'get', image_id)
  File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 219, in call
    return getattr(client.images, method)(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/glanceclient/v2/images.py", line 177, in get
    return self.model(**body)
  File "/usr/lib/python2.7/site-packages/warlock/core.py", line 32, in __init__
    base_class.__init__(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/warlock/model.py", line 34, in __init__
    raise ValueError(str(exc))
ValueError: None is not of type u'string'

Failed validating u'type' in schema[u'additionalProperties']:
    {u'type': u'string'}

On instance[u'ramdisk_id']:
    None

J-PMethot (jpmethot) wrote :

This issue may be linked to this https://bugs.launchpad.net/nova/+bug/1467570 . Basically, I have just been confirmed that Nova provisions instances from rbd by querying glanceV2. While it affects the code for ephemeral rbd snapshots, as said earlier, this also affect the code for provisioning instances in Kilo and prevents VMs from booting from snapshot when using a rbd backend.

Hence, a backport of this bug fix for kilo would be strongly appreciated.

Ian Cordasco (icordasc) on 2015-07-02
Changed in glance:
status: In Progress → Invalid
Samuel Matzek (smatzek) wrote :

This is purely a Glance bug and can be recreated without Nova in the picture at all, therefore I do not believe that the bug fix mentioned in comment 5 will fix this.

See the recreation steps in comment 2. This can be completely abstracted from Nova. The bug is that Glance v1 allows you to set properties with no value, while Glance v2 uses schema validation and validates that those same 2 properties MUST have a string value.

After that point, once you bring Nova into the picture, snapshot images created with libvirt in Nova will have an issue once Nova moves to use Glance v2 for all image access. This is because kernel_id and ramdisk_id can be set to no value on images created in earlier releases using Glance v1 and the Glance image list/ show APIs will fail on those images.

Changed in glance:
status: Invalid → Confirmed
Stuart McLaren (stuart-mclaren) wrote :
Download full text (3.2 KiB)

I installed a stable kilo devstack, but still wasn't able to get the image-list reproducer to work:

$ glance --version
0.17.2

$ glance --os-image-api-version 1 image-create --name test --disk-format raw --container-format bare --property ramdisk_id= --property kernel_id= --file /etc/fstab
+-----------------------+--------------------------------------+
| Property | Value |
+-----------------------+--------------------------------------+
| Property 'kernel_id' | None |
| Property 'ramdisk_id' | None |
| checksum | 9cb02fe7fcac26f8a25d6db3109063ae |
| container_format | bare |
| created_at | 2015-07-23T15:55:41.000000 |
| deleted | False |
| deleted_at | None |
| disk_format | raw |
| id | a139ed67-e254-44d7-9499-78b13b498bb4 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | test |
| owner | 3447cea05d6947658d73791ed9e0ed9f |
| protected | False |
| size | 145 |
| status | active |
| updated_at | 2015-07-23T15:55:42.000000 |
| virtual_size | None |
+-----------------------+--------------------------------------+

$ glance --os-image-api-version 2 image-list
+--------------------------------------+---------------------------------+
| ID | Name |
+--------------------------------------+---------------------------------+
| 75fb76bc-cb1c-49b0-a821-704a3525dfcc | cirros-0.3.2-x86_64-uec |
| 9325073e-fc51-494f-b80b-f9de075dc00f | cirros-0.3.2-x86_64-uec-kernel |
| 9cf53d72-8b1b-4f94-9ecc-6c2061c74034 | cirros-0.3.2-x86_64-uec-ramdisk |
| e156d634-3087-4361-a6ad-89fbf544ae2e | cirros-0.3.4-x86_64-uec |
| f0134c5f-1563-4700-8f33-880334a5f834 | cirros-0.3.4-x86_64-uec-kernel |
| d22d2060-3b6c-44fa-823b-e7d108b43d9d | cirros-0.3.4-x86_64-uec-ramdisk |
| a139ed67-e254-44d7-9499-78b13b498bb4 | test |
+--------------------------------------+---------------------------------+

I'm struggling to understand why I'm not able to reproduce using image-list. I do suspect that the image-list issue has gone away with an updated client, but can't be 100% sure.

The nova stack trace in comment #4 is from an image-show rather than an image list.

I am able to reproduce using image-show (as suggested by Mike Fedosin) ie

$ glance --os-image-api-version 1 image-create --name test --disk-format raw --container-format bare --property ramdisk_id= --property kernel_id=

$ glance --os-image-api-version 2 image-show b51e04cc...

Read more...

Reviewed: https://review.openstack.org/176379
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=82c63b1df4399db22dfd71e3326a6bfc1b372a9c
Submitter: Jenkins
Branch: master

commit 82c63b1df4399db22dfd71e3326a6bfc1b372a9c
Author: Jorge Niedbalski <email address hidden>
Date: Wed Apr 22 12:54:29 2015 -0300

    Allow ramdisk_id, kernel_id to be null on schema

    Since is possible to create images with no
    ramdisk_id, kernel_id parameters on glance v1,
    then the schema-image.json needs to be modified to
    allow null, string values for both attributes
    and prevent LP #1447193 to happen with a glance v2
    client.

    APIImpact

    Closes-Bug: #1447215
    Related-Bug: #1447193

    Change-Id: I645c797553ccaa84062f9ad28f2349ae738a0c96
    Signed-off-by: Jorge Niedbalski <email address hidden>

Changed in glance:
status: Confirmed → Fix Committed
Changed in glance:
status: Fix Committed → Fix Released
Louis Taylor (kragniz) on 2015-07-27
Changed in glance:
importance: Undecided → High
Changed in glance (Ubuntu Vivid):
status: New → Confirmed
Changed in glance (Ubuntu):
status: Confirmed → Fix Committed
status: Fix Committed → Confirmed

Reviewed: https://review.openstack.org/205432
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=4b269775964f75300d3af56a4c57234d54a32e94
Submitter: Jenkins
Branch: stable/kilo

commit 4b269775964f75300d3af56a4c57234d54a32e94
Author: Jorge Niedbalski <email address hidden>
Date: Wed Apr 22 12:54:29 2015 -0300

    Allow ramdisk_id, kernel_id to be null on schema

    Since is possible to create images with no
    ramdisk_id, kernel_id parameters on glance v1,
    then the schema-image.json needs to be modified to
    allow null, string values for both attributes
    and prevent LP #1447193 to happen with a glance v2
    client.

    APIImpact

    Closes-Bug: #1447215
    Related-Bug: #1447193

    Change-Id: I645c797553ccaa84062f9ad28f2349ae738a0c96
    Signed-off-by: Jorge Niedbalski <email address hidden>
    (cherry picked from commit 82c63b1df4399db22dfd71e3326a6bfc1b372a9c)

Jorge Niedbalski (niedbalski) wrote :
Changed in glance (Ubuntu Vivid):
status: Confirmed → In Progress
assignee: nobody → Jorge Niedbalski (niedbalski)
tags: added: patch sts

The attachment "Vivid Patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Changed in glance (Ubuntu):
status: Confirmed → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glance - 1:2015.1.1-0ubuntu2

---------------
glance (1:2015.1.1-0ubuntu2) vivid; urgency=medium

  * Additional support for stable/kilo (LP: #1481008):
    - d/p/fix-test-artifacts-plugin-loader.patch: Added to fix failing tests.
    - d/p/skip-glance-search-tests.patch: Added to fix failing tests.
    - d/p/skip-online-tests.patch: Added to skip tests that require online access
      during Ubuntu builds.

 -- Corey Bryant <email address hidden> Wed, 05 Aug 2015 17:53:11 -0400

Changed in glance (Ubuntu Vivid):
status: In Progress → Fix Released
James Page (james-page) on 2015-09-02
Changed in glance (Ubuntu):
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers