OpenStack Compute (nova)

image-create does not respect the force_raw_images setting

Bug #1497484 reported by Nicolas Simonds
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Nicolas Simonds
Mitaka
Fix Released
Medium
Lee Yarwood

Bug Description

Instance snapshots of instances sourced from, e.g., QCOW2 images will be created in the image service as "qcow2" and then switched to "raw" in an update step.

Use case:

We decided to drop QCOW2 support from certain product configurations, as force_raw_images is enabled by default, and the conversion overhead made for a sub-wonderful customer experience.

After dropping QCOW2 from the acceptable list of image formats from Glance, clients could no longer make instance snapshots from instances that were spawned from QCOW2 images, despite the fact that the backing store was not QCOW2.

Steps to Reproduce:

1. Upload a QCOW2 image into Glance
2. Update Nova/Glance configs to disable QCOW2 images and enable force_raw_images
3. Boot an instance against the QCOW2 image
4. Create a snapshot of the instance

Expected behavior:

A snapshot of the instance

Actual results:
    ERROR (BadRequest): <html>
     <head>
      <title>400 Bad Request</title>
     </head>
     <body>
      <h1>400 Bad Request</h1>
      Invalid disk format 'qcow2' for image.<br /><br />

     </body>
    </html> (HTTP 400) (HTTP 400) (Request-ID: req-8e8d8d51-8e0c-4033-bb84-774d2ed1f90a)

Markus Zoeller (markus_z) (mzoeller)
tags: added: config-options snapshots
tags: added: glance
tags: added: snapshot
removed: snapshots
tags: added: images
removed: glance
Revision history for this message
Nicolas Simonds (nicolas.simonds) wrote :

Addendum on the reproduction steps:

Step 2 involves configuring Nova as:

    [DEFAULT]
    force_raw_images = True
    use_cow_images = False

...and Glance as:

    [DEFAULT]
    disk_formats = ami,ari,aki,raw,iso

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/226069

Changed in nova:
assignee: nobody → Nicolas Simonds (nicolas.simonds)
status: New → In Progress
Revision history for this message
Matthew Booth (mbooth-9) wrote :

The reported error appears to originate from glance. Without logs it's hard to be 100% certain, but my supposition is that it's raised when nova tries to push the created snapshot to glance, and glance rejects it because it's qcow2.

The question then becomes: why is nova creating a qcow2 snapshot image? The reasons would be appear to be either that the existing disk is qcow2 (or was mis-detected as such, see bug 1524274), or that the config variable snapshot_image_format is set to qcow2. Or alternatively some other tortured reason in this organic heap of code.

This does suggest a solution, though. If you set 'snapshot_image_format' to 'raw' in nova.conf, it should always convert to raw before uploading to glance. If my original supposition is correct, this should fix your problem without any code changes.

Revision history for this message
Nicolas Simonds (nicolas.simonds) wrote :

Nova is not creating a QCOW2 snapshot; it's creating a RAW snapshot, and telling Glance that it created a QCOW2 snapshot. Glance actually detects the mismatch, and updates the disk_format after ingest. So the whole "Nova lies to Glance" family of bugs is known about on their side, and has been worked around already.

Unless, of course, you tell Glance not to accept QCOW2 images for ingest. Then it does a full-stop and the whole thing flies apart.

Revision history for this message
Nicolas Simonds (nicolas.simonds) wrote :

I was able to reproduce the bug quite easily given the recommended configuration; the error occurs in the Nova API, and none of the other services seem to log anything untoward happening at all:

    2016-01-14 22:23:55.831 DEBUG nova.api.openstack.wsgi [req-1ad0dc6d-5021-4d6c-8f92-b4333e61849e admin admin] Action: 'action', calling method: <bound method ServersController._action_create_image of <nova.api.openstack.compute.servers.ServersController object at 0x7fd2a9c5b450>>, body: {"createImage": {"name": "zomg", "metadata": {}}} from (pid=15657) _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:795
    2016-01-14 22:23:55.979 DEBUG nova.compute.api [req-1ad0dc6d-5021-4d6c-8f92-b4333e61849e admin admin] [instance: c3a69c63-e0e7-4b58-aadc-51e53ba2478d] Fetching instance by UUID from (pid=15657) get /opt/stack/nova/nova/compute/api.py:1946
    2016-01-14 22:23:56.086 INFO nova.api.openstack.wsgi [req-1ad0dc6d-5021-4d6c-8f92-b4333e61849e admin admin] HTTP exception thrown: 400 Bad Request: Invalid disk format 'qcow2' for image. (HTTP 400)
    2016-01-14 22:23:56.087 DEBUG nova.api.openstack.wsgi [req-1ad0dc6d-5021-4d6c-8f92-b4333e61849e admin admin] Returning 400 to user: 400 Bad Request: Invalid disk format 'qcow2' for image. (HTTP 400) from (pid=15657) __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1180
    2016-01-14 22:23:56.090 INFO nova.osapi_compute.wsgi.server [req-1ad0dc6d-5021-4d6c-8f92-b4333e61849e admin admin] 10.0.2.15 "POST /v2.1/1b03f2f440e74a8191d8d62027c1e460/servers/c3a69c63-e0e7-4b58-aadc-51e53ba2478d/action HTTP/1.1" status: 400 len: 402 time: 0.2918901

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

Tagged with an official tag.

tags: added: config
removed: config-options
Sahid Orentino (sahid-ferdjaoui)
tags: added: liberty-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/226069
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8cfc8a837592c9494608d68db2bac1954c573a4b
Submitter: Jenkins
Branch: master

commit 8cfc8a837592c9494608d68db2bac1954c573a4b
Author: Nicolas Simonds <email address hidden>
Date: Mon Sep 21 11:17:43 2015 -0700

    Compute API: omit disk/container formats when creating images of snapshots

    If making a snapshot, omit the disk and container formats from the
    image API request, since the image may have been converted to another
    format than what the base image uses. The driver will correctly
    populate these values later when uploading the image data.

    Closes-Bug: 1497484
    Change-Id: I69522977631be2ae42ac504740c04247c75022eb

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/322084

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote : Fix included in openstack/nova 14.0.0.0b1

This issue was fixed in the openstack/nova 14.0.0.0b1 development milestone.

Matt Riedemann (mriedem)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/mitaka)

Reviewed: https://review.openstack.org/322084
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0c21aff7e1624b6877b2e4289d8d19700adc9b8d
Submitter: Jenkins
Branch: stable/mitaka

commit 0c21aff7e1624b6877b2e4289d8d19700adc9b8d
Author: Nicolas Simonds <email address hidden>
Date: Mon Sep 21 11:17:43 2015 -0700

    Compute API: omit disk/container formats when creating images of snapshots

    If making a snapshot, omit the disk and container formats from the
    image API request, since the image may have been converted to another
    format than what the base image uses. The driver will correctly
    populate these values later when uploading the image data.

    Closes-Bug: 1497484
    Change-Id: I69522977631be2ae42ac504740c04247c75022eb
    (cherry picked from commit 8cfc8a837592c9494608d68db2bac1954c573a4b)

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/nova 13.1.0

This issue was fixed in the openstack/nova 13.1.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.