upload-to-image doesn't work with the Ceph RBD backend

[weiguo sun]

Hi all,

The command (on grizzly):

$ cinder upload-to-image --force True --container-format bare --disk-format raw 60595635-b4a6-42de-a2d0-488c2ac75115 new-test-image
ERROR: The server has either erred or is incapable of performing the requested operation.

The cinder-api.log:

2013-07-01 12:06:12 ERROR [cinder.api.middleware.fault] Caught error: 'Controller' object has no attribute 'create'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/fault.py", line 73, in __call__
    return req.get_response(self.application)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__
    return self.app(env, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
    response = self.app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
    return resp(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 803, in __call__
    content_type, body, accept)
  File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 851, in _process_stack
    action_result = self.dispatch(meth, request, action_args)
  File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 927, in dispatch
    return method(req=request, **action_args)
  File "/usr/lib/python2.7/dist-packages/cinder/api/contrib/volume_actions.py", line 180, in _volume_upload_image
    force)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 63, in wrapped
    return func(self, context, target_obj, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/cinder/volume/api.py", line 717, in copy_volume_to_image
    recv_metadata = self.image_service.create(context, metadata)
  File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 245, in create
    **sent_service_image_meta)
  File "/usr/lib/python2.7/dist-packages/cinder/image/glance.py", line 146, in call
    return getattr(client.images, method)(*args, **kwargs)
AttributeError: 'Controller' object has no attribute 'create'

[Haomai Wang (haomai)]

run command:
cinder upload-to-image --force True --container-format bare --disk-format raw 349cbcdf-070d-443c-8084-a3daaff562c5 new-test-image

It seems no error when running command.

Can you provide more info?

[weiguo sun (wsun2)]

Sure, here is our setup,

cinder & glance use the same ceph cluster backend but separate pool for glance and cinder as you can see from the config file below. CephX is enabled with the same key for both pools. For the openstack general setup,

(1) All services running on this controller node with two extra nova nodes total.
(2) All normal cinder/glance/nova functions regarding rbd work as expected, including live-migration, cinder boot volume creation from glance image linked-clone, snapshot of the cinder volume, nova booting from cinder volume and volume snapshot, cinder volume creation from another cinder volume snapshot, glance image creation to the backend rbd pool from a flat file, etc.

Here are the config files.

> cinder.conf
[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
sql_connection = mysql://cinderUser:cinderPass@10.33.156.153/cinder
#iscsi_helper = tgtadm
iscsi_helper = ietadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
#osapi_volume_listen_port=5900
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_pool=svl-ceph-openstack-volumes
rbd_user=svl-ceph-openstack-volumes
rbd_secret_uuid=e7724cad-23f6-d4ce-8c68-5f40491b15dd
glance_api_version=2

> glance.conf
.....
default_store = rbd
.....
rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_user=svl-ceph-openstack-images
rbd_store_pool=svl-ceph-openstack-images
rbd_store_chunk_size = 8
show_image_direct_url=True

> dpkg output for the cinder/controller host,

oot@svl-ceph-10:/etc/glance# cat /etc/motd
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.5.0-27-generic x86_64)

ii ceph-common 0.61.4-1precise common utilities to mount and interact with a ceph storage cluster
ii cinder-api 1:2013.1-0ubuntu1~cloud0 Cinder storage service - API server
ii cinder-common 1:2013.1-0ubuntu1~cloud0 Cinder storage service - common files
ii cinder-scheduler 1:2013.1-0ubuntu1~cloud0 Cinder storage service - Scheduler server
ii cinder-volume 1:2013.1-0ubuntu1~cloud0 Cinder storage service - Volume server
ii glance 1:2013.1-0ubuntu1~cloud0 OpenStack Image Registry and Delivery Service - Daemons
ii glance-api 1:2013.1-0ubuntu1~cloud0 OpenStack Image Registry and Delivery Service - API
ii glance-common 1:2013.1-0ubuntu1~cloud0 OpenStack Image Registry and Delivery Service - Common
ii glance-registry 1:2013.1-0ubuntu1~cloud0 OpenStack Image Registry and Delivery Service - Registry
rc libcephfs1 0.56.4-1precise Ceph distributed file system client library
ii python-ceph 0.61.4-1precise ...

[weiguo sun (wsun2)]

Hi Haomai,
I updated the bug with some of our configuration information. Not sure if there is anything you would like to see. I am on IRC now. Did see you pop in and out quickly earlier. If you are still availabe to chat on IRC for our problem, that will be great.
Thanks. --weiguo

> Date: Wed, 3 Jul 2013 02:27:21 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1196666] Re: upload-to-image doesn't work with the Ceph RBD backend
>
> run command:
> cinder upload-to-image --force True --container-format bare --disk-format raw 349cbcdf-070d-443c-8084-a3daaff562c5 new-test-image
>
> It seems no error when running command.
>
> Can you provide more info?
>
>
> ** Changed in: cinder
> Status: Confirmed => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1196666
>
> Title:
> upload-to-image doesn't work with the Ceph RBD backend
>
> Status in Cinder:
> Incomplete
>
> Bug description:
> Hi all,
>
> The command (on grizzly):
>
> $ cinder upload-to-image --force True --container-format bare --disk-format raw 60595635-b4a6-42de-a2d0-488c2ac75115 new-test-image
> ERROR: The server has either erred or is incapable of performing the requested operation.
>
> The cinder-api.log:
>
> 2013-07-01 12:06:12 ERROR [cinder.api.middleware.fault] Caught error: 'Controller' object has no attribute 'create'
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/fault.py", line 73, in __call__
> return req.get_response(self.application)
> File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
> application, catch_exc_info=False)
> File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
> app_iter = application(self.environ, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> return resp(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__
> return self.app(env, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> return resp(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> return resp(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
> response = self.app(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> return resp(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
> resp = self.call_func(req, *args, **self.kwargs)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
> return self.func(req, *args, **kwargs)
> File "/usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py", line 803, in __call__
> content_type, body, accept)
> File "/usr/lib/python2.7/dist-packages/cinder/api/openstac...

[weiguo sun (wsun2)]

Turning on the debug for cinder with the log files.

[Haomai Wang (haomai)]

I'm sorry that when you added comments and I'm asleep, we are in different time zone.

I debug the command and insert the probe to image/glance.py, it seems that your client exists some errors.

I suggest that you can verify your class and correspond attributes when debug. If you can provide more debug info, the problem will be easily solved.

> /opt/stack/cinder/cinder/image/glance.py(149)call()
(Pdb) p client
<glanceclient.v1.client.Client object at 0x408d890>
(Pdb) p client.images
<glanceclient.v1.images.ImageManager object at 0x408d9d0>
(Pdb) p dir(client.images)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_delete', '_format_image_meta_for_user', '_get_file_size', '_image_meta_from_headers', '_image_meta_to_headers', '_list', '_update', 'api', 'create', 'data', 'delete', 'get', 'list', 'resource_class', 'update']
(Pdb) p client.images
<glanceclient.v1.images.ImageManager object at 0x408d9d0>

[weiguo sun (wsun2)]

No problem, which timezone are you based? Beijing? Any suggestions how I can get more debugging information? The debug=True is already turned for cinder.conf as well as verbose=True.
BTW, I am on IRC now and maybe we can chat quickly if you are online?
Thx for the help. --weiguo
> Date: Thu, 4 Jul 2013 03:20:26 +0000
> From: <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1196666] Re: upload-to-image doesn't work with the Ceph RBD backend
>
> I'm sorry that when you added comments and I'm asleep, we are in
> different time zone.
>
> I debug the command and insert the probe to image/glance.py, it seems
> that your client exists some errors.
>
> I suggest that you can verify your class and correspond attributes when
> debug. If you can provide more debug info, the problem will be easily
> solved.
>
> > /opt/stack/cinder/cinder/image/glance.py(149)call()
> (Pdb) p client
> <glanceclient.v1.client.Client object at 0x408d890>
> (Pdb) p client.images
> <glanceclient.v1.images.ImageManager object at 0x408d9d0>
> (Pdb) p dir(client.images)
> ['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_delete', '_format_image_meta_for_user', '_get_file_size', '_image_meta_from_headers', '_image_meta_to_headers', '_list', '_update', 'api', 'create', 'data', 'delete', 'get', 'list', 'resource_class', 'update']
> (Pdb) p client.images
> <glanceclient.v1.images.ImageManager object at 0x408d9d0>
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1196666
>
> Title:
> upload-to-image doesn't work with the Ceph RBD backend
>
> Status in Cinder:
> Incomplete
>
> Bug description:
> Hi all,
>
> The command (on grizzly):
>
> $ cinder upload-to-image --force True --container-format bare --disk-format raw 60595635-b4a6-42de-a2d0-488c2ac75115 new-test-image
> ERROR: The server has either erred or is incapable of performing the requested operation.
>
> The cinder-api.log:
>
> 2013-07-01 12:06:12 ERROR [cinder.api.middleware.fault] Caught error: 'Controller' object has no attribute 'create'
> Traceback (most recent call last):
> File "/usr/lib/python2.7/dist-packages/cinder/api/middleware/fault.py", line 73, in __call__
> return req.get_response(self.application)
> File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
> application, catch_exc_info=False)
> File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in call_application
> app_iter = application(self.environ, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> return resp(environ, start_response)
> File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 450, in __call__
> return self.app(env, start_response)
> File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> return resp(environ, start_response)
> ...

[weiguo sun (wsun2)]

please see the pdb output from glance.py. It looks like that glanceclient.v2 was called instead of v1 from your example. --weiguo

(Epdb) p client
<glanceclient.v2.client.Client object at 0x2e713d0>
(Epdb) p client.images
<glanceclient.v2.images.Controller object at 0x2ba7c50>
(Epdb) p dir(client.images)
['__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', 'data', 'delete', 'get', 'http_client', 'list', 'model', 'update']
(Epdb) p client.images
<glanceclient.v2.images.Controller object at 0x2ba7c50>

After disabling "glance_api_version=2" in cinder.conf, the error dissappears and the image got registered in glance.

However, the new image creation stuck at "queued" status,

72fe1a63-6d40-4dcd-ba68-a85745218c29 | new-test-image | raw | bare | | queued |

[weiguo sun (wsun2)]

It turns out that the latest issue is due to that I didn't restart cinder-volume service after disabling "glance_api_version=2". After restarting cinder-volume service, the upload-to-image is working.

In summary, the issue lies in that the confliction when setting "glance_api_version=2" in cinder.conf in order to get benefit of copy-on-write cloning for cinder volume creation. But Glance API v2 has not implement "create" function which leads to the failure of upload-to-image function.

This issue will force people to pick one of the two features unless they are resorting to some complex multi cinder configuration setup.

[Huang Zhiteng (zhiteng-huang)]

Looks like a configuration issue rather than a bug. I'll close it. Thanks for following up, Weiguo.

[weiguo sun (wsun2)]

Hi Zhiteng,

Can we keep this open? I believe this should be categorized as a bug. Given that "glance_api_version=2" is valid configuration in cinder.conf to enable COW cloning feature for rbd cinder volume store, I see this bug will prevent one to implement two cinder functions ("COW cloning" vs "upload-to-image") concurrently.

Let me know if this doesn't make sense.

Thanks. --weiguo

[Haomai Wang (haomai)]

@weiguo You means that if we disable "glance_api_version=2" in cinder.conf, it will show the error? But I think it's the must thing if you want to use ceph backend.

[Huang Zhiteng (zhiteng-huang)]

Weiguo, sorry I failed to see the relation between glance_api_version and CoW cloning feature in RBD based on my read on Cinder's code. Would you please elaborate?

[weiguo sun (wsun2)]

Haomai> I mean the other way around, ie, turning off "glance_api_version=2" will make the error go away and "upload-to-image" will work. It is a must thing for CoW cloning feature to function with ceph backend. It is not a must if you don't need the CoW cloning when creating cinder volumes.

Zhiteng> From the debug output, I can see glance_api_version 2 didn't implement image_create call. When you execute the "upload-to-image" from cinder CLI, it calls glance_api to create/register image in glance, with version 2 glance_api, it will fail since the create function is not yet implemented. Hopefully this make sense.

[weiguo sun (wsun2)]

Haimai> when I stated "It is a must thing for CoW cloning feature to function with ceph backend.", I intended to say that enabling "glance_api_version=2" is a must thing for CoW cloning feature with ceph backend. Just to clarify.

[Huang Zhiteng (zhiteng-huang)]

Weiguo, Cinder is using 'update' call after a 'create' call to glance to upload image, no matter what version of glance api is using. And from api.openstack.org and the glance source code, I don't think there's any API missing in glance v2 API.

[weiguo sun (wsun2)]

Hmm, maybe glance-client v2 does not implement "create" function and cinder is calling glance-client v2? See the following command output, which indicate no "create function". If you reset the API_VERSION to 1, you can see the difference.

root@svl-ceph-10:~# export OS_IMAGE_API_VERSION="2"
root@svl-ceph-10:~# glance --help
usage: glance [--version] [-d] [-v] [-k] [--cert-file CERT_FILE]
              [--key-file KEY_FILE] [--os-cacert <ca-certificate-file>]
              [--ca-file OS_CACERT] [--timeout TIMEOUT] [--no-ssl-compression]
              [-f] [--dry-run] [--ssl] [-H ADDRESS] [-p PORT]
              [--os-username OS_USERNAME] [-I OS_USERNAME]
              [--os-password OS_PASSWORD] [-K OS_PASSWORD]
              [--os-tenant-id OS_TENANT_ID] [--os-tenant-name OS_TENANT_NAME]
              [-T OS_TENANT_NAME] [--os-auth-url OS_AUTH_URL] [-N OS_AUTH_URL]
              [--os-region-name OS_REGION_NAME] [-R OS_REGION_NAME]
              [--os-auth-token OS_AUTH_TOKEN] [-A OS_AUTH_TOKEN]
              [--os-image-url OS_IMAGE_URL] [-U OS_IMAGE_URL]
              [--os-image-api-version OS_IMAGE_API_VERSION]
              [--os-service-type OS_SERVICE_TYPE]
              [--os-endpoint-type OS_ENDPOINT_TYPE] [-S OS_AUTH_STRATEGY]
              <subcommand> ...

Command-line interface to the OpenStack Images API.

Positional arguments:
  <subcommand>
    explain Describe a specific model.
    image-delete Delete specified image.
    image-download Download a specific image.
    image-list List images you can access.
    image-show Describe a specific image.
    help Display help about this program or one of its
                        subcommands.

Optional arguments:

[weiguo sun (wsun2)]

Hi Zhiteng,

At this point, I believe it is that glanceclient v2 didn't implement create function. See the following difference between v1 and v2

root@svl-ceph-10:~# grep create /usr/share/pyshared/glanceclient/v2/images.py
root@svl-ceph-10:~#
root@svl-ceph-10:~# grep create /usr/share/pyshared/glanceclient/v1/images.py
                   'size', 'id', 'created_at', 'updated_at')
    def create(self, **kwargs):
                msg = 'create() got an unexpected keyword argument \'%s\''

There is no create defined in v2 version of the glanceclient, which cinder uses when performing "upload-to-image".

--weiguo

[Haomai Wang (haomai)]

Thank you for weiguo!

I have reproduced the error you said, your elaborate on explanation is great!

[Zhi Yan Liu (lzy-dev)]

I consider wsun2 is correct, from code checking and shell example output result. Actually I have realized this situation before, but under my understanding in nova code, I'm sure the version number can be set as an argument in some glance related functions, but I was not checking the cinder code before. Also I will try to get the historic reason that why v2 client does not be implemented fully as v1 client in python-glanceclient.

[Iccha Sethi (iccha-sethi)]

Hey glanceclient work for create in v2 is in review https://review.openstack.org/#/c/35504/

[Haomai Wang (haomai)]

It's strange that glance client leave out such important implementation.
I will focus on it and see whether the bug is really fixed by the python-glanceclient

[Edward Hope-Morley (hopem)]

This issue has hopefully been resolved by https://bugs.launchpad.net/cinder/+bug/1210467. Marking as Duplicate for now since this seems to be the same issue and a fix has been released. If it this is still a problem please re-open.