find_flavor doesn't work for names

[Matthew Edmonds]

The fix that was committed for https://bugs.launchpad.net/python-openstacksdk/+bug/1384230 appears not to have worked, or to have been broken again by a subsequent change. Using openstacksdk 0.9.5, I can find a flavor by id:

(Pdb) conn.compute.find_flavor('72d1d0cf-4f0d-4736-895b-abfe62f13651')
openstack.compute.v2.flavor.Flavor(name=test, links=[{u'href': u'https://myhostname:8774/v2.1/2d4c094a598d4888851d8db0a170f33e/flavors/72d1d0cf-4f0d-4736-895b-abfe62f13651', u'rel': u'self'}, {u'href': u'https://myhostname:8774/2d4c094a598d4888851d8db0a170f33e/flavors/72d1d0cf-4f0d-4736-895b-abfe62f13651', u'rel': u'bookmark'}], ram=1024, OS-FLV-DISABLED:disabled=False, vcpus=1, swap=, os-flavor-access:is_public=True, rxtx_factor=1.0, OS-FLV-EXT-DATA:ephemeral=0, disk=0, id=72d1d0cf-4f0d-4736-895b-abfe62f13651)

but I cannot find by name:

(Pdb) conn.compute.find_flavor('test')
*** NotFound: Not Found (HTTP 404)

[Brian Curtin (brian.curtin)]

Odd. That works for me

>>> #f is a flavor I picked out of conn.compute.flavors()
>>> f.name
'm1.tiny'
>>> f.id
'1'
>>> conn.compute.find_flavor(f.name)
openstack.compute.v2.flavor.Flavor(id=1, name=m1.tiny, ...other stuff...)
>>> conn.compute.find_flavor(f.id)
openstack.compute.v2.flavor.Flavor(id=1, name=m1.tiny, ...other stuff...)

Can you enable debug logging to a file and then try this again and send it to me? http://developer.openstack.org/sdks/python/openstacksdk/users/utils.html is the logging method details, and you can either host it somewhere and link to it, or email <email address hidden> and I'll take a look.

[Matthew Edmonds (edmondsw)]

Thanks, Brian. This is the only information in the logs, aside from authentication:

2016-09-29 14:34:25,413 DEBUG: keystoneauth.session REQ: curl -g -i --cacert "/etc/pki/tls/certs/powervc.crt" -X GET https://myhostname:8774/v2.1/2d4c094a598d4888851d8db0a170f33e/flavors/test -H "User-Agent: test keystoneauth1/2.12.1 python-requests/2.10.0 CPython/2.7.5" -H "X-Auth-Token: {SHA1}2b2d31dcc68e055d4f0515f5b46402f8172f024d"
2016-09-29 14:34:25,564 DEBUG: keystoneauth.session RESP: [404] Date: Thu, 29 Sep 2016 18:34:25 GMT Server: Apache Openstack-Api-Version: compute 2.1 X-Openstack-Nova-Api-Version: 2.1 Vary: OpenStack-API-Version,X-OpenStack-Nova-API-Version Content-Type: application/json; charset=UTF-8 Content-Length: 77 X-Compute-Request-Id: req-056a5fb3-a031-4304-8a22-e2fadcb80c19 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive
RESP BODY: {"itemNotFound": {"message": "Flavor test could not be found.", "code": 404}}

2016-09-29 14:34:25,564 DEBUG: keystoneauth.session Request returned failure status: 404

[Brian Curtin (brian.curtin)]

This is weird. That small piece makes sense, as the list method first tries to do a GET based on what you give it—-in this case a name which will 404 since the server is expecting an id--but it should then then be doing a second GET for all items and then look through the results to find a flavor that matches the name.

Doing the same thing yields this log for me: http://paste.openstack.org/show/583543/, and this chunk of code here is what's being hit: https://github.com/openstack/python-openstacksdk/blob/master/openstack/resource2.py#L788

One curious thing is in your initial message you have "NotFound: Not Found (HTTP 404)", which is not something that should be showing up. For one, find_flavor should just be returning None if it doesn't find something that matches, not raising an exception. The second is that exception looks like it's coming out of keystoneauth, whereas we've wrapped the KSA session for a few small things, including an exception mapper to openstacksdk exception types.

Are you able to share a bit more of your code here or elsewhere? This seems a bit odd. If I had to guess, are you creating a Session direct from keystoneauth and passing it in when you create your Connection instance? That might explain why the KSA exception is evading our attempt at handling it for a separate retry, and why it would have shown up.

[Matthew Edmonds (edmondsw)]

I see the problem... The exception that's being raised is:

Traceback (most recent call last):
  File "./test", line 26, in <module>
    f = conn.compute.find_flavor('test')
  File "/usr/lib/python2.7/site-packages/openstack/compute/v2/_proxy.py", line 66, in find_flavor
    ignore_missing=ignore_missing)
  File "/usr/lib/python2.7/site-packages/openstack/proxy2.py", line 105, in _find
    **attrs)
  File "/usr/lib/python2.7/site-packages/openstack/resource2.py", line 790, in find
    return match.get(session)
  File "/usr/lib/python2.7/site-packages/openstack/resource2.py", line 590, in get
    response = session.get(request.uri, endpoint_filter=self.service)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 667, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/positional/__init__.py", line 94, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneauth1/session.py", line 570, in request
    raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.NotFound: Not Found (HTTP 404)

but the exception that's being expected and handled (realizing it's not an id, and moving on to find it by name) in openstack.resource2.find is openstack.exceptions.NotFoundException... similar name, but not the same thing.

I threw a breakpoint in openstack.session.map_exceptions, which appears to be the only place that raises NotFoundException, and it never triggered in this flow. Instead of using openstack.session, it appears to be using keystoneauth1.session... and then it hit me... This is how I created my session:

import argparse
import sys
from keystoneauth1 import loading as ks_loading
from openstack import connection
parser = argparse.ArgumentParser()
ks_loading.register_auth_argparse_arguments(parser, sys.argv[1:],default='v3password')
ks_loading.register_session_argparse_arguments(parser)
args = parser.parse_args()
auth = ks_loading.load_auth_from_argparse_arguments(args)
sess = ks_loading.load_session_from_argparse_arguments(args, auth=auth)
conn = connection.Connection(session=sess, authenticator=auth)

And that works for other things... but not here, because of the apparent expectation that the session is an instance of openstack.session.Session. But I don't think I can do what I've been doing (creating a session based on OS_* environment variables or --os-* CLI arguments) if I switch to an openstack.session.Session. And I expect this will trip up others who don't realize there are two different types of sessions and they can't use the very widely used keystoneauth1 version. Even though it actually seems to work for a while.

[Brian Curtin (brian.curtin)]

What if rather than doing this all yourself, you used something like openstack.connection.from_config, which uses os-client-config and takes care of what I think is the same stuff you're trying to get done?

I guess we could have some other thing that converts a KSA session into that openstacksdk wrapper class. We have that wrapper because KSA is (somewhat, in a way) an implementation detail and we don't want to have to require users to be involved with or know about KSA or the exceptions it's raising. What you're doing should work, but it should also work how it has been with KSA being behind the scenes.

[Matthew Edmonds (edmondsw)]

openstack.connection.from_config doesn't let you pull from environment variables though, does it? I've already got environment variables set (as most folks do, since those are used for the CLI commands). I don't already have a config file setup with the same information.

[Brian Curtin (brian.curtin)]

I don't use that method for anything myself, but it apparently takes an argparse namespace: https://github.com/openstack/python-openstacksdk/blob/master/openstack/connection.py#L75, and os-client-config works with all of the usual environment variables: http://docs.openstack.org/developer/os-client-config/

[Matthew Edmonds (edmondsw)]

Thanks for your help, Brian. The suggestion in comment 7 didn't work, but it got me going in the right direction, and I've opened separate bugs for the problems that kept it from working.

So as for this defecdt... I agree that find_flavor works, and this was user error. I think the question that remains is whether we just invalidate this bug or use it to make a change so that others aren't opening similar bugs in the future. That change would probably be to detect that a keystoneauth session was used and throw a nice error back. You could wrapper the session and keep going, but you might hit more issues... If we don't need to support this because there are better ways to satisfy the argparse usecase, as there appear to be, I probably wouldn't invest in allowing keystoneauth sessions to be used here.

[Brian Curtin (brian.curtin)]

I'll take a look and see if we can detect a true KSA session and just convert it into the one we need rather than erroring out. We should be able to satisfy both cases of just giving creds and using our own session or taking other sessions, I think.

[Brian Curtin (brian.curtin)]

I'm starting to think that's not actually realistic to convert between the two classes since it'd just be copying some KSA Session instance attributes over to construct a SDK Session, which would maybe be close but likely flaky.

We have some overrides of the base KSA Session that we need in order to work as expected, like with endpoint selection, so I think we're going to have to need this to always be an SDK Session coming through. I think we can get what you need by just improving that from_config to generate the proper things coming out of os-client-config, whether they be from a YAML file or env vars.

[Matthew Edmonds (edmondsw)]

I agree with comment 10. As I said in comment 8, I think you'd be better off raising an error if someone passes in a keystoneauth1 session than trying to convert it. Make them give you an SDK session... just fail more nicely when they don't. As it stands today, some things work, and some things don't, and it's not obvious why... All avoidable if it initially checked, saw that it was a keystoneauth1 session, and stopped you right there. I'll propose something.

[Matthew Edmonds (edmondsw)]

proposed https://review.openstack.org/#/c/380495/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/380495
Committed: https://git.openstack.org/cgit/openstack/python-openstacksdk/commit/?id=4614979e7e49518c66812b233fbe6389bfc86ee3
Submitter: Jenkins
Branch: master

commit 4614979e7e49518c66812b233fbe6389bfc86ee3
Author: Matthew Edmonds <email address hidden>
Date: Fri Sep 30 15:01:30 2016 -0400

    avoid usage of keystoneauth1 sessions

    Passing a keystoneauth1 session to the Connection constructor
    instead of an openstacksdk session leads to strange errors. This
    change will check for this usage error and raise a clear exception.

    Change-Id: I5d2d74502ef8abcb6c609864ce6fd5f5ec55af2a
    Closes-Bug: #1628316