UUID case sensitivity causing unfriendly and unexpected API errors

[Ryan Tidwell]

I was seeing a user pass an uppercase UUID from their app to the Nova API's. Their calls failed with a 500 error and the following exception in the nova-api log:

017-05-24 11:20:27.176 68631 ERROR nova.api.openstack [req-674b4fd1-9a22-488e-97d2-29fca9ca3b7c 98afbc5675e059f7b1dde34c44d615712e3aafed8afcdfb5d890276ec554e737 951955d46e3a45c99e4c8649e8abe247 - - -] Caught error: 'NoneType' object has no attribute 'get'
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack Traceback (most recent call last):
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 139, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack return req.get_response(self.application)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack application, catch_exc_info=False)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack return resp(environ, start_response)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack response = req.get_response(self._app)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1317, in send
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack application, catch_exc_info=False)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1281, in call_application
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack return resp(environ, start_response)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack return resp(environ, start_response)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack response = self.app(environ, start_response)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack return resp(environ, start_response)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 672, in __call__
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack content_type, body, accept)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in _process_stack
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack request, action_args)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 619, in post_process_extensions
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack **action_args)
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/hide_server_addresses.py", line 63, in show
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack self._perhaps_hide_addresses(instance, resp.obj['server'])
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/hide_server_addresses.py", line 52, in _perhaps_hide_addresses
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack if instance.get('vm_state') in self.hide_address_states:
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack AttributeError: 'NoneType' object has no attribute 'get'
2017-05-24 11:20:27.176 68631 ERROR nova.api.openstack
2017-05-24 11:20:27.177 68631 INFO nova.api.openstack [req-674b4fd1-9a22-488e-97d2-29fca9ca3b7c 98afbc5675e059f7b1dde34c44d615712e3aafed8afcdfb5d890276ec554e737 - - -] http://api.d1cp1.ttc.openstack.target.com:8774/v2.1/<>/servers/5100C340-8967-4DB1-97F7-BFA1C155F250 returned with HTTP 500

Having the user pass lowercase UUID's in their app fixes the client issue, but it would be nice if the API's would sanitize UUID's or at least kick out a more friendly error.

[Ryan Tidwell (ryan-tidwell)]

I forgot to mention I'm seeing this on stable/mitaka, but looks like it could still be an issue on master.

[Matt Riedemann (mriedem)]

The problem is the hide_server_addresses extension is looking for the instance in the request cache using the id passed in via the client, which is upper case, and the id key in the cache is from the db, which is lower-case, so it doesn't find the instance and blows up with a TypeError because it's not checking for None. This is something you'll hit with mysql which is case insensitive so it can find the instance but doesn't cache it using the same key.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/467772

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/467772
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ecfb65cee68e96cc99ceecddbf27c6d6f768358e
Submitter: Zuul
Branch: master

commit ecfb65cee68e96cc99ceecddbf27c6d6f768358e
Author: Matt Riedemann <email address hidden>
Date: Wed May 24 17:06:35 2017 -0400

    Fix instance lookup in hide_server_addresses extension

    The hide_server_addresses extension is looking up the cached
    instance based on what the user provided for the server id,
    which may not match what is used to cache the instance for the
    request. For example, a request with upper-case server uuid
    could be found in a mysql-backed system because mysql is
    case insensitive by default, but the instance is keyed off the
    server id from the DB, which is lower-case, so we'll fail
    to look up the instance in the cache if the IDs don't match.

    There is no test for this because it turns out it's actually
    really hard to recreate this since it requires running with a
    mysql backend to recreate the case insensitive check, which
    isn't going to work with sqlite. Given how trivial this fix is,
    creating a big mysql recreate test is not worth it.

    Change-Id: I09b288aa2ad9969800a3cd26c675b002c6c9f638
    Closes-Bug: #1693335

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/518996

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/518997

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/518996
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=31b2f0a2a65b782be4d0a68b3b53604a076f4748
Submitter: Zuul
Branch: stable/pike

commit 31b2f0a2a65b782be4d0a68b3b53604a076f4748
Author: Matt Riedemann <email address hidden>
Date: Wed May 24 17:06:35 2017 -0400

    Fix instance lookup in hide_server_addresses extension

    The hide_server_addresses extension is looking up the cached
    instance based on what the user provided for the server id,
    which may not match what is used to cache the instance for the
    request. For example, a request with upper-case server uuid
    could be found in a mysql-backed system because mysql is
    case insensitive by default, but the instance is keyed off the
    server id from the DB, which is lower-case, so we'll fail
    to look up the instance in the cache if the IDs don't match.

    There is no test for this because it turns out it's actually
    really hard to recreate this since it requires running with a
    mysql backend to recreate the case insensitive check, which
    isn't going to work with sqlite. Given how trivial this fix is,
    creating a big mysql recreate test is not worth it.

    Change-Id: I09b288aa2ad9969800a3cd26c675b002c6c9f638
    Closes-Bug: #1693335
    (cherry picked from commit ecfb65cee68e96cc99ceecddbf27c6d6f768358e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.0 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/518997
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=09970fe366cf3200a53d93c41bc996a58a25a4af
Submitter: Zuul
Branch: stable/ocata

commit 09970fe366cf3200a53d93c41bc996a58a25a4af
Author: Matt Riedemann <email address hidden>
Date: Wed May 24 17:06:35 2017 -0400

    Fix instance lookup in hide_server_addresses extension

    The hide_server_addresses extension is looking up the cached
    instance based on what the user provided for the server id,
    which may not match what is used to cache the instance for the
    request. For example, a request with upper-case server uuid
    could be found in a mysql-backed system because mysql is
    case insensitive by default, but the instance is keyed off the
    server id from the DB, which is lower-case, so we'll fail
    to look up the instance in the cache if the IDs don't match.

    There is no test for this because it turns out it's actually
    really hard to recreate this since it requires running with a
    mysql backend to recreate the case insensitive check, which
    isn't going to work with sqlite. Given how trivial this fix is,
    creating a big mysql recreate test is not worth it.

    Change-Id: I09b288aa2ad9969800a3cd26c675b002c6c9f638
    Closes-Bug: #1693335
    (cherry picked from commit ecfb65cee68e96cc99ceecddbf27c6d6f768358e)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.1 release.