"kernel_id property not found" Openstack API w/ S3ImageService

[Dan Prince]

Using 2011.1~bzr638-

When using the default nova.image.s3.S3ImageService I am unable to create an image using the Openstack API. The following exception is reported in nova-api.log:

2011-01-28 15:53:07,207 ERROR nova.api.openstack [-] Caught error: kernel_id property not found for image ami-tty
(nova.api.openstack): TRACE: Traceback (most recent call last):
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", line 52, in __call__
(nova.api.openstack): TRACE: return req.get_response(self.application)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
(nova.api.openstack): TRACE: application, catch_exc_info=False)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
(nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
(nova.api.openstack): TRACE: response = self.app(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
(nova.api.openstack): TRACE: return resp(environ, start_response)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
(nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
(nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func

Using the same setup I am able to create an image with the EC2 API via eucatools.

[Dan Prince (dan-prince)]

I've push if fix for this issue here:

  lp:~dan-prince/nova/openstack_api_709355

It appears that the default S3 image store does not require kernel_id or ramdisk_id to be set. This patch allows that and makes the openstack API behave like the EC2 API.

[Dan Prince (dan-prince)]

Oops.
WRong project.

[Dan Prince (dan-prince)]

Please delete this and I'll repost in nova.

[Jay Pipes (jaypipes)]

Hi Dan! Project changed :)

But, I don't see why the openstack API should be following the path of the EC2 API and eucatools? Could you explain why this is necessary?

AFAIK, for the OpenStack API, we're trying to get away from the awkward euca-bundle-image API that requires setting up kernel, ramdisk, and machine images separately...

Thanks!
jay

[Dan Prince (dan-prince)]

My goal here was to create a new server via a POST request to the Openstack API using a default installation (single node). Since the default image store uses the S3 Image store I got the above exception.

Aren't we trying to support a setup where tools can use both the Openstack and EC2 API's at the same time? In my case it just seemed odd that the EC2 API worked find (when using euca-tools) but the Openstack API required the extra parameters to be defined in the image store.

Is there some other way I can work around this?

[Jay Pipes (jaypipes)]

Hi!

"Aren't we trying to support a setup where tools can use both the Openstack and EC2 API's at the same time?"

eucatools don't work with the OpenStack API. Never have and never will :) The cloudservers programs work with the OpenStack API.

If you are running XenServer, I believe you can set the Nova flag --xenapi_image_service to the nova.image.s3, or something like that, to get XenServer to use nova-objectstore (which speaks the S3 API) as an image service instead of Glance...

I'm going to try to grab Rick Harris to see if he can elaborate with better details than my limited knowledge can offer.

[Vish Ishaya (vishvananda)]

This seems resonable, OS_API should work by default. Your patch removes i18n though, you'll need to switch it back to using _() and dict based replacement strings. It should probably also be more specific, meaning other missing values should probably still error.

On Jan 28, 2011, at 9:29 AM, Dan Prince wrote:

> My goal here was to create a new server via a POST request to the
> Openstack API using a default installation (single node). Since the
> default image store uses the S3 Image store I got the above exception.
>
> Aren't we trying to support a setup where tools can use both the
> Openstack and EC2 API's at the same time? In my case it just seemed odd
> that the EC2 API worked find (when using euca-tools) but the Openstack
> API required the extra parameters to be defined in the image store.
>
> Is there some other way I can work around this?
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/709355
>
> Title:
> "kernel_id property not found" Openstack API w/ S3ImageService
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Using 2011.1~bzr638-
>
> When using the default nova.image.s3.S3ImageService I am unable to
> create an image using the Openstack API. The following exception is
> reported in nova-api.log:
>
>
> 2011-01-28 15:53:07,207 ERROR nova.api.openstack [-] Caught error: kernel_id property not found for image ami-tty
> (nova.api.openstack): TRACE: Traceback (most recent call last):
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", line 52, in __call__
> (nova.api.openstack): TRACE: return req.get_response(self.application)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
> (nova.api.openstack): TRACE: application, catch_exc_info=False)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
> (nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
> (nova.api.openstack): TRACE: response = self.app(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
> (no...

[Dan Prince (dan-prince)]

Sure. That would be great.

To be clear I wasn't trying to use euca-tools with the Openstack API though. I'm actually using the Ruby Openstack binding with the following code:

require 'openstackcompute'

USERNAME=ENV['CLOUD_SERVERS_USERNAME']
API_KEY=ENV['CLOUD_SERVERS_API_KEY']
API_URL=ENV['CLOUD_SERVERS_URL']
cs = OpenStackCompute::Connection.new(:username => USERNAME, :api_key => API_KEY, :api_url => API_URL)
cs.create_server(:name => "Dan", :imageId => 9076875775488977738, :flavorId => 1)

--

My nova.conf is below if your interested:

--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--verbose
--my_ip=172.19.0.3
--public_interface=eth1
--vlan_interface=eth1
--iscsi_ip_prefix=172.19.0
--sql_connection=mysql://nova:zzzz@172.19.0.3/nova
--libvirt_type=qemu
--rabbit_host=172.19.0.3
--rabbit_port=5672
--rabbit_userid=nova
--rabbit_password=eRmSm0Ss2eZhJrmRjrpz
--rabbit_virtual_host=/nova
--network_manager=nova.network.manager.FlatManager
--default_project=admin

---

So using the Ruby Openstack binding with the Openstack API I hit this exception with my setup.

I only used euca-tools because I wanted to verify provisioning with my setup was in fact working.

[Vish Ishaya (vishvananda)]

I don't believe he is trying to use eucatools. He's trying to create an image via the os api using nova.image.s3.S3ImageService (which is the default). There is some oddness in the traceback though, it appears that ami-tty doesn't have a kernel_id and ramdisk_id specified, although I believe it should have one (ari_tty, aki_tty).

Vish

On Jan 28, 2011, at 9:40 AM, Jay Pipes wrote:

> Hi!
>
> "Aren't we trying to support a setup where tools can use both the
> Openstack and EC2 API's at the same time?"
>
> eucatools don't work with the OpenStack API. Never have and never will
> :) The cloudservers programs work with the OpenStack API.
>
> If you are running XenServer, I believe you can set the Nova flag
> --xenapi_image_service to the nova.image.s3, or something like that, to
> get XenServer to use nova-objectstore (which speaks the S3 API) as an
> image service instead of Glance...
>
> I'm going to try to grab Rick Harris to see if he can elaborate with
> better details than my limited knowledge can offer.
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/709355
>
> Title:
> "kernel_id property not found" Openstack API w/ S3ImageService
>
> Status in OpenStack Compute (Nova):
> New
>
> Bug description:
> Using 2011.1~bzr638-
>
> When using the default nova.image.s3.S3ImageService I am unable to
> create an image using the Openstack API. The following exception is
> reported in nova-api.log:
>
>
> 2011-01-28 15:53:07,207 ERROR nova.api.openstack [-] Caught error: kernel_id property not found for image ami-tty
> (nova.api.openstack): TRACE: Traceback (most recent call last):
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", line 52, in __call__
> (nova.api.openstack): TRACE: return req.get_response(self.application)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
> (nova.api.openstack): TRACE: application, catch_exc_info=False)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
> (nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
> (nova.api.openstack): TRACE: response = self.app(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): ...

[Jay Pipes (jaypipes)]

Ah, sorry, I was misunderstanding. Sorry, Dan.

/me didn't even know there was a Ruby OpenStack binding...

I'll ping back on this later...heading to airport right now.

-jay

On Fri, Jan 28, 2011 at 11:54 AM, Vish Ishaya <email address hidden> wrote:
> I don't believe he is trying to use eucatools.  He's trying to create an
> image via the os api using nova.image.s3.S3ImageService (which is the
> default).  There is some oddness in the traceback though, it appears
> that ami-tty doesn't have a kernel_id and ramdisk_id specified, although
> I believe it should have one (ari_tty, aki_tty).
>
> Vish
>
> On Jan 28, 2011, at 9:40 AM, Jay Pipes wrote:
>
>> Hi!
>>
>> "Aren't we trying to support a setup where tools can use both the
>> Openstack and EC2 API's at the same time?"
>>
>> eucatools don't work with the OpenStack API. Never have and never will
>> :) The cloudservers programs work with the OpenStack API.
>>
>> If you are running XenServer, I believe you can set the Nova flag
>> --xenapi_image_service to the nova.image.s3, or something like that, to
>> get XenServer to use nova-objectstore (which speaks the S3 API) as an
>> image service instead of Glance...
>>
>> I'm going to try to grab Rick Harris to see if he can elaborate with
>> better details than my limited knowledge can offer.
>>
>> --
>> You received this bug notification because you are a member of Nova Bug
>> Team, which is subscribed to OpenStack Compute (nova).
>> https://bugs.launchpad.net/bugs/709355
>>
>> Title:
>>  "kernel_id property not found" Openstack API w/ S3ImageService
>>
>> Status in OpenStack Compute (Nova):
>>  New
>>
>> Bug description:
>>  Using 2011.1~bzr638-
>>
>>  When using the default nova.image.s3.S3ImageService I am unable to
>>  create an image using the Openstack API. The following exception is
>>  reported in nova-api.log:
>>
>>
>>  2011-01-28 15:53:07,207 ERROR nova.api.openstack [-] Caught error: kernel_id property not found for image ami-tty
>>  (nova.api.openstack): TRACE: Traceback (most recent call last):
>>  (nova.api.openstack): TRACE:   File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", line 52, in __call__
>>  (nova.api.openstack): TRACE:     return req.get_response(self.application)
>>  (nova.api.openstack): TRACE:   File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
>>  (nova.api.openstack): TRACE:     application, catch_exc_info=False)
>>  (nova.api.openstack): TRACE:   File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
>>  (nova.api.openstack): TRACE:     app_iter = application(self.environ, start_response)
>>  (nova.api.openstack): TRACE:   File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
>>  (nova.api.openstack): TRACE:     return resp(environ, start_response)
>>  (nova.api.openstack): TRACE:   File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
>>  (nova.api.openstack): TRACE:     return resp(environ, start_response)
>>  (nova.api.openstack): TRACE:   File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
>>  (nova.api.openstack): TRACE:     return resp(environ, start_response)
>>  (nova.api.opens...

[Dan Prince (dan-prince)]

Okay. I pushed to openstack_api_709355 again and I think I fixed the i18n issue.

[Thierry Carrez (ttx)]

This is not really the recommended combination. We can try to fix it and if we come up with a really contained fix it might be considered for Bexar. Otherwise I wouldn't take the risk and just document that you should use Glance with the OS API.

[Dan Prince (dan-prince)]

Sure. Thanks Thierry. I'm looking into a solution to this now (any comments/help appreciated!)

I totally understand that the Openstack API will work best with Glance. Glance however isn't currently the default image store for the project so it would seem that we would also want to support other image stores as well. My use case here was simple to be able to test Openstack API ruby bindings with a default nova installation.

My initial openstack_api_709355 branch fix was to change a private function called 'lookup' to log a DEBUG error instead of raising an exception. This function is only used to lookup 'kernel_id' and 'ramdisk_id' and is all contained in the _get_kernel_ramdisk_from_image so it shouldn't affect anything else.

On second look a better solution might be to conditionally handle the different image stores. I'm not familar with Glance yet (looking at that now) but it appears the image objects from Glance and S3 are somewhat different:

according to the comment on _get_kernel_ramdisk_from_image Glance uses:

  image[''image_properties']['kernel_id']

where the S3 image store uses the following:

  image[''kernelId']

--

I'm not in a rush to have this for Bexar BTW. This issue however would appear to prevent anyone from using the Openstack API with anything but Glance.

Dan

[Thierry Carrez (ttx)]

> This issue however would appear to prevent anyone from using the Openstack API with anything but Glance.

You can still use the Openstack API to start instances successfully stored in S3ImageService. The OpenstackAPI -> S3ImageService path is mostly untested, so I fear as soon as this bug is fixed we'll hit another one. Better have it completely tested and supported in Cactus that risking a regression three days from release. You can still propose a very elegant patch that convinces me otherwise, though :)

[Dan Prince (dan-prince)]

Okay. It appears that GlanceImageService and S3ImageService return image dictionaries that have different keys. The following patch is an unclean solution that allows me to use both S3ImageService and GlanceImageService via the Openstack API.

---

         def lookup(param):
- _image_id = image_id
- try:
+
+ def _underscore_to_camelcase(str):
+ x=''.join([x[:1].upper() + x[1:] for x in str.split('_')])
+ y=x[0].lower()+x[1:]
+ return y
+
+ _image_id = image_id
+ try:
                 return image['properties'][param]
- except KeyError:
- LOG.debug(_("Param %s not defined for image_id %s."), param, image_id)
+ except KeyError:
+ try:
+ return image[_underscore_to_camelcase(param)]
+ except:
+ raise exception.NotFound(_("%(param)s property not found for image %(_image_id)s") % locals())

----

The idea here is to try what Glance is looking for first and then try the S3 key. If both fail the exception is thrown.

Like I said above this works but seems to be kind of dirty. Are there any plans on making the image services standardize on an interface for image dictionaries? Should we make Glance match S3 or vice versa?

[Dan Prince (dan-prince)]

15:34 < dprince> jaypipes: So my question is really about whether there is a
                 way to standardize the image service dictionaries so that both
                 API controllers work cleanly?
15:34 < jaypipes> dprince: yes, there is :) just not sure whether devcamcar
                  already coded up something or whether to use your patch :)
15:38 < jaypipes> dprince: gonna have a quick call with sirp_ and then I'll
                  figure it out :)

---

I'll hold on working this out until I hear back more. I'm happy to work up a cleaner solution but I'd like some guidance first on whether we want to change Glance or S3 to match the interface we choose.

[Dan Prince (dan-prince)]

Okay. After a some commits from others (thanks Devin) I've worked up what I think is a good solution to this issue. This includes:

-Make S3 and Glance adhere to a common interface.
-Remove S3 image object mapping code from the Openstack images API.
-Remove custom kernel_id and ramdisk_id from the Openstack images API (this is
 done via the compute API). Per ticket 709355 (this ticket).
-Updated the base image service docs.

Submitting the merge request now....

[Dan Prince (dan-prince)]

This bug was arguably fixed by Ricks commit in 750. I think Justin also worked around it by returning None, None in a similar revision as well.

My proposal for this fix was to promote 'kernel_id' and 'ramdisk_id' to top level image object attributes and then let the compute API handle it (which it already does). Lots of stuff changing within the image services for now so I'll save my changes for later.

Marking this as fixed and moving on.