Unable to upload volume data to image on Unified SDK

[Thiago Carreira]

This feature is available and works on old SDK (Cinder), but does not on Unified SDK. The image service have an "upload_image" method (on unified SDK) but I cannot get the data to upload it, as the data is stored as a volume in the block store service, and the only way to download it is through glance - which is what I am trying to achieve. Here follows some code sample:

---------------
UNIFIED SDK
---------------

from openstack import connection

import glanceclient.v2.client as glance_client
from keystoneclient.auth.identity import v3
from keystoneauth1 import session
import cinderclient.v3.client as cinder_client

import sys
import os
import time
import getopt
import json
import subprocess

def get_connection_to_services(auth_url, username, password, user_domain_name, project_name, project_domain_name):
    conn = connection.Connection(
        auth_url=auth_url,
        project_name=project_name,
        project_domain_name=project_domain_name,
        username=username,
        user_domain_name=user_domain_name,
        password=password)
    return conn

inputfile = 'credentials.json'
with open(inputfile) as json_file:
    json_string = json_file.read()
    json_obj = json.loads(json_string)

username = json_obj["username"]
password = json_obj["password"]
auth_url_sdk = json_obj["auth_url_sdk"]
user_domain_name = json_obj["user_domain_name"]
project_domain_name = json_obj["project_domain_name"]
project_name = json_obj["project_name"]

conn = get_connection_to_services(
                auth_url=auth_url_sdk,
                project_name=project_name,
                project_domain_name=project_domain_name,
                username=username,
                user_domain_name=user_domain_name,
                password=password)
compute_service = conn.compute
network_service = conn.network
image_service = conn.image
block_service = conn.block_store

# id of desired volume to upload to glance.
id = 'e5b40f95-0e24-46d0-a405-6759b4167f97'

# type of volume: openstack.block_store.v2.volume.Volume
volume = block_service.get_volume(id)

#cinder volume url
data = 'http://10.32.135.74:8776/v2/8136b8ab6da346ed8c342022a91aa975/volumes/e5b40f95-0e24-46d0-a405-6759b4167f97'

# this throws an error
image_service.upload_image('bare', 'raw', volume)

# instead of volume, if we use data, it upload the url string bytes
image_service.upload_image('bare', 'raw', data)

------------
Cinder SDK
------------

def get_session(auth_url, username, password, user_domain_name, project_name, project_domain_name):
    auth = v3.Password(
        auth_url=auth_url,
        username=username,
        password=password,
        user_domain_name=user_domain_name,
        project_name=project_name,
        project_domain_name=project_domain_name)
    return session.Session(auth=auth)

old_session = get_session(
                    auth_url_sdk,
                    username,
                    password,
                    user_domain_name,
                    project_name,
                    project_domain_name)

cinder = cinder_client.Client(session=old_session)

# type of volume2 = cinderclient.v3.volumes.Volume
volume2 = cinder.volumes.get(id)
body = {'force': 'false','image_name': 'volume2', 'container_format': 'bare', 'disk_format': 'raw'}
# works like a charm :D
volume2.upload_to_image(**body)

[Brian Curtin (brian.curtin)]

I'm not sure what the problem is. If you give it data to upload, it will upload it, as expected.

You'll also need to include actual error messages or logs if you think there's a problem.

[Brian Curtin (brian.curtin)]

get_volume(id) gets a volume resource, not the contents of the volume. Perhaps we need to add a download_volume(id) method that gets the data contained in the volume?

[Thiago Carreira (tcanascimento)]

Thanks for the anwser.

By debuging the OpenStack CLI, we see that it makes the following POST:

---------
GET call to volumev2 for http://10.32.135.74:8776/v2/8136b8ab6da346ed8c342022a91aa975/volumes/e15028ca-513c-46d7-a2a4-5cdac67ec0ce used request id req-eaa70503-2f77-4ab9-b1ec-0881c9c91721
REQ: curl -g -i -X POST http://10.32.135.74:8776/v2/8136b8ab6da346ed8c342022a91aa975/volumes/e15028ca-513c-46d7-a2a4-5cdac67ec0ce/action -H "User-Agent: python-cinderclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}2260f8172113503603e00751d55494bb45abbdff" -d '{"os-volume_upload_image": {"container_format": "bare", "force": false, "image_name": "Tyrell", "disk_format": "raw"}}'
"POST /v2/8136b8ab6da346ed8c342022a91aa975/volumes/e15028ca-513c-46d7-a2a4-5cdac67ec0ce/action HTTP/1.1" 202 628
RESP: [202] X-Compute-Request-Id: req-f7f9623e-948b-4429-9d32-569449100c48 Content-Type: application/json Content-Length: 628 X-Openstack-Request-Id: req-f7f9623e-948b-4429-9d32-569449100c48 Date: Fri, 14 Oct 2016 16:39:35 GMT Connection: keep-alive
----------

Response:

----------
RESP BODY: {"os-volume_upload_image": {"status": "uploading", "container_format": "bare", "image_name": "Tyrell", "updated_at": "2016-10-14T14:45:39.000000", "image_id": "98ac5f2e-3672-46c7-8baf-698fb970bc03", "display_description": "", "is_public": false, "id": "e15028ca-513c-46d7-a2a4-5cdac67ec0ce", "size": 1, "disk_format": "raw", "volume_type": {"description": null, "deleted": false, "created_at": "2016-10-07T20:58:03.000000", "updated_at": null, "extra_specs": {"volume_backend_name": "lvmdriver-1"}, "is_public": true, "deleted_at": null, "id": "2b0cec41-5050-4fe2-acf5-c81d5bdb0aa3", "name": "lvmdriver-1"}, "protected": false}}
----------

In none of the API version documentation would one find this 'os-volume_upload_image' object (http://developer.openstack.org/api-ref/block-storage/v3/). However, taking a blink on the openstackclient code (https://github.com/openstack/python-openstackclient/blob/276675f3520fa707daa65f92219194a514623fb8/openstackclient/image/v2/image.py), which API calls for, we find out that API calls the cinder 'upload_to_image' method - and there appears the 'os-volume_upload_image' parameter as an object key for volume properties.

So, there is three suggestions:

(i) a method for download volume directly from Cinder would be nice if it does not come against OpenStack features and policies.
(ii) update the unified SDK considering Cinder client methods.
(iii) update the API documentation

[Brian Curtin (brian.curtin)]

We need (i) regardless of this. We already have upload/download calls in Object Store and Image, and we should have the same for Block Store. If we had it, I think you'd be ok.

If I knew more about 'os-volume_upload_image' we could potentially add it straight into the SDK calls. I don't have time to read up much at the moment, but googling for it only returned things in the mailing list, and the first thing I saw said it might depend on a policy setting. If we can depend on it, perhaps we can add support for it.

[Thiago Carreira (tcanascimento)]

Ok, thanks.
We will keep working on it. Any new we report it here.