nova-api should handle type conversion while creating server snapshots

[Dilip Renkila]

Hi all,

We are running on rocky, when we create a snapshot we are hitting at the following error.

dilip@ThinkPad-T430:~$ openstack server image create --name backupc_snap backuppc

400 Bad Request: Provided object does not match schema 'image': u'False' is not of type 'boolean': Failed validating 'type' in schema['properties']['os_hidden']:: {'description': u'If true, image will not appear in default image list response.',: 'type': 'boolean'}: On instance['os_hidden']:: u'False' (HTTP 400) (HTTP 400) (Request-ID: req-e19827a6-9e43-4eca-89bd-92fecf05557f)

AFAIK , glance recently introduced os_hidden boolean attribute to image by default and its not being converted to correct type by nova api

the log file is below

root@ctrl2:~# tail -f /var/log/nova/nova-api.log | grep ERROR
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi [req-f93abf09-ed22-45c7-972e-1d6f0c0e3343 ff9650c86533492581513eca72b48409 2eea218eea984dd68f1378ea21c64b83 - 765703fcca634b149c7a012626847d2f 765703fcca634b149c7a012626847d2f] Unexpected exception in API method: TypeError: Unable to set 'os_hidden' to 'False'. Reason: u'False' is not of type u'boolean'
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 801, in wrapped
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 110, in wrapper
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py", line 1041, in _action_create_image
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi extra_properties=metadata)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 213, in _wrapped
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi return fn(self, context, instance, *args, **kwargs)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 153, in inner
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi return f(self, context, instance, *args, **kw)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2815, in snapshot
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi extra_properties=extra_properties)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 2888, in _create_image
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi return self.image_api.create(context, image_meta)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 118, in create
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi return session.create(context, image_info, data=data)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 460, in create
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi data, force_activate)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 537, in _create_v2
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi context, 2, 'create', **sent_service_image_meta)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 179, in call
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi result = getattr(controller, method)(*args, **kwargs)
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/glanceclient/common/utils.py", line 545, in inner
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi return RequestIdProxy(wrapped(*args, **kwargs))
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/glanceclient/v2/images.py", line 281, in create
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi raise TypeError(encodeutils.exception_to_unicode(e))
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi TypeError: Unable to set 'os_hidden' to 'False'. Reason: u'False' is not of type u'boolean'
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi Failed validating u'type' in schema[u'properties'][u'os_hidden']:
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi {u'description': u'If true, image will not appear in default image list response.',
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi u'type': u'boolean'}
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi On instance[u'os_hidden']:
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi u'False'
2018-12-02 13:20:15.446 3377353 ERROR nova.api.openstack.wsgi

[Matt Riedemann (mriedem)]

One workaround might be to add "os_hidden" to the non_inheritable_image_properties configuration option to filter that property out when creating the snapshot:

https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.non_inheritable_image_properties

[Matt Riedemann (mriedem)]

The problem is likely in this code:

https://github.com/openstack/nova/blob/c9dca64fa64005e5bea327f06a7a3f4821ab72b1/nova/image/glance.py#L734

Where we're not handling os_hidden and converting it to a boolean.

[Matt Riedemann (mriedem)]

There must be some other way this gets enabled because otherwise we should have seen this fail in Tempest when the os_hidden property was added to glance images - does the server have to expose the field or something? Does os_hidden need to be true for this to fail?

Can you give specific steps for how to reproduce this?

[Dilip Renkila (dilip-renkila278)]

Hi all,

steps to reproduce it

1. nova-api version - 18.0.0
2. glance client version- 2.9.1
3. create an instance with an image which has the property os_hidden = false.
4. try to create a snapshot of that instance

[Dilip Renkila (dilip-renkila278)]

full trace file https://etherpad.openstack.org/p/e20ERvBEkw

[Matt Riedemann (mriedem)]

Isn't os_hidden=false in all images by default since Rocky? I can't understand why tempest wouldn't hit this in it's snapshot testing, unless it's something with glanceclient in the mix - nova uses glanceclient but tempest doesn't. Tempest has its own REST client for glance. I'll see if I can reproduce in devstack.

[Matt Riedemann (mriedem)]

Using devstack the os_hidden property is set in the default image:

http://paste.openstack.org/show/737226/

I had no problem creating a snapshot from that image:

http://paste.openstack.org/show/737229/

I'm using glanceclient:

$ pip show python-glanceclient
Name: python-glanceclient
Version: 2.15.0

Based on this release note in glanceclient:

https://github.com/openstack/python-glanceclient/commit/2b33e6858a669426092ddeb210b6d1d21fa257f9

You might need >=2.12.0?

https://review.openstack.org/#/c/579117/

[Matt Riedemann (mriedem)]

I downgraded glanceclient to version 2.9.1 and re-did the snapshot and it still worked.

[Matt Riedemann (mriedem)]

Granted, I'm using a devstack created today:

stack@stein:~/devstack$ git log -1
commit 8fc227f63fd0820bd9c5725a6023529c742d2041
Merge: 2697e43 5a9e32a
Author: Zuul <email address hidden>
Date: Wed Dec 12 23:29:07 2018 +0000

    Merge "Update comment about nova-cells-v1 job"
stack@stein:~/devstack$

So maybe I need to try and recreate this from a stable/rocky devstack, but note that in stable/rocky devstack would still install a newer python-glanceclient than you're using because it uses upper-constraints:

https://github.com/openstack/requirements/blob/stable/rocky/upper-constraints.txt#L102

[Balazs Gibizer (balazs-gibizer)]

@Dilip: Could you reproduce the same problem with python-glanceclient >= 2.12.0 [1]?

I'm setting the bug to Incomplete as we are missing something that prevents us to reproduce the issue. Please set the bug status back to New if you have more information.

[1] https://github.com/openstack/python-glanceclient/commit/2b33e6858a669426092ddeb210b6d1d21fa257f9

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]