glance-cache-manage talks to image endpoint instead of the specified host

[Sasikiran]

Corrected bug description (based on the comments below):
In Glance, image cache is local to each node. And, hence, image cache management must be performed on each node locally. This is supported by providing a specific host using '-H' or '--host' argument to glance-cache-manage utility.

However, due to a bug glance-cache-manage currently ignores the provided host and fetches the image endpoint to manage the cache. While this does manage the cache, we don't know what node is being managed here. This can be fixed by passing 'configure_via_auth=False' when creating the image cache client. This ensures that image endpoint is not fetched and the provided host is used.

Initial bug description:
If we want glance-cache-manage commands to communicate over internalURL, it doesn't support. It always use publicURL for communication.

[Dharini Chandrasekar (dharini-chandrasekar)]

@Sasikiran: Can you please tell us what you exactly mean by telling glance-cache-manage is not communicating over the internal url?
Did you mean for communication via keystone?
If you intend to use the internalURL for glance-cache-manage, you can set it using --os-auth-url along with glance-manage-cache. The store specific endpoints are used by Glance to communicate with the backend.. That does not set glance-cache-manage to use it as cache is cleared directly from glance.
Thanks.

[Sasikiran (sasikiran-vaddi)]

@Dharini: I want to use the endpoint internalURL for glance service but not keystone.

Let me elaborate.

Around 216 line of glance/common/auth.py. Please find the below code snippet.

if self.configure_via_auth:
    endpoint = get_endpoint(resp_auth['serviceCatalog'],
                            endpoint_region=creds_region)
    self.management_url = endpoint

Here there is no possibility for the user to pass endpoint_type as a parameter for the method get_endpoint() to retrieve the internalURL of Glance. By default it takes 'publicURL'.

I have an env# where I need to use glance endpoint as internalURL but not publicURL. But glance-cache-manage doesn't have an option to pass internalURL

Here the issue is

1) glance commands are working fine as I can export the following variable GLANCE_ENDPOINT_TYPE=internalURL
2) But glance-cache-manage is not working for me. Do I need to do any configurations changes or does glance codebase doesn't have support for it? I have checked the command line options for glance-cache-manage, but I haven't find any paramter which I can specify endpoint_type.

[Dharini Chandrasekar (dharini-chandrasekar)]

Hello Sasikiran,

The endpoint you set using the env variable OS_AUTH_URL is the keystone endpoint to use for authentication by any service. So glance service will talk to that url (whatever you have set it to - internal/public/admin) every time it needs to. So when you set glance's endpoint to use internalURL, it means glance is talking to other services including keystone with its internal url.
glance-cache-manage would be picking up keystone's internal url to talk to as you have set it as your env variable. (By default, glance-cache-manage will talk to whatever is given for OS_AUTH_URL)
So if your glance-cache-manage command does not go through you need to check if your glance's endpoint that you have set as intenralURL (GLANCE_ENDPOINT_TYPE=internalURL) has it registered with keystone. Your other Glance commands might have worked because it was picking up the publicURL of glance to talk to keytsone's internalURL.

[Kairat Kushaev (kkushaev)]

So the problem is:
1) we cannot use internalURL from keystone service catalog in glance manage cache. manage cache client always connecting to public glance URL which may be not appropriate in some cases (especially when we have different functionality for different endpoint types).
2) there should be possibility which glance endpoint to choose from keystone service catalog.
Here is the patch for this bug: https://review.openstack.org/#/c/391859/

[Ian Cordasco (icordasc)]

Sasikiran, can you explain what "glance-cache-manage is not working for me" means? Are you getting exceptions? What does "not working" even mean? We need more details to be able to verify we've fixed this for you.

[Sasikiran (sasikiran-vaddi)]

@Ian:

When we pass internalURL for glance-cache-manage by exporting environment variable "GLANCE_ENDPOINT_TYPE=internalURL",
it doesn't get the endpoint for internalURL from glance, rather it gets public endpoint.

The code snippet from the below link:

https://github.com/openstack/glance/blob/stable/mitaka/glance/common/auth.py#L215-L217

Here while retrieving endpoint for glance service, there is no feasibility to pass endpoint_type, so it fetches by default endpoint_type which is publicURL.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/396620

[Ian Cordasco (icordasc)]

Sasikiran, you're not describing what you expect to happen when using the internal URL that is not happening. I understand it's not using the internal URL. But what would be different if it did? What behaviour are you seeing right now that's causing issues?

[Hemanth Makkapati (hemanth-makkapati)]

Sasikiran,
I understand what you are saying about glance-cache-manage commands not using internalURL. And, you are right that it always defaults to publicURL. However, glance-cache-manage commands should never work with any endpoint in the first place, be it public/internal/admin.

In Glance, image cache is local to each node. Therefore, cache management should be performed explicitly on each node. Not via an endpoint which may or may not hit the node you wish to manage cache on. So, I think it is more appropriate for us to fix the glance-cache-manage commands to not hit the endpoint at all. This way, you can specifically hit the node you want and on whatever interface you desire.

Let me know if this makes sense.
I'm going to change the description of this bug to reflect what I just said.

Here is the patch that fixes the issue I described above.
https://review.openstack.org/#/c/393229/

[Ian Cordasco (icordasc)]

So this bug does affect newton and earlier, but newton is presently the only stable branch that is not restricted to critical and security fixes. This isn't a terribly critical bug, in my evaluation, and as such I've only nominated this to be backported to newton.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/393229
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=616060db65ac3f12c0a71fd21f8156c6e993d7ab
Submitter: Jenkins
Branch: master

commit 616060db65ac3f12c0a71fd21f8156c6e993d7ab
Author: kairat_kushaev <email address hidden>
Date: Thu Nov 3 13:21:05 2016 +0300

    Do not use service catalog for cache client

    Previously cache client used endpoints from service catalog to
    request cache items from node. It works perfectly on devstack
    but it doesn't work when an OpenStack cloud is deployed with HA
    (3/5/7 controllers). We need to use -H option instead.
    This option makes cache management client interact with the local
    endpoint which is more appropriate as image cache is local to
    each node/controller.

    Change-Id: I355fb36d9af7e06dcac70b8263dc1a2b915ffc8f
    Closes-Bug: #1634486

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/glance 14.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/396620
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=d9e584bd4fd95cdb19b89d0edf591c13c10cd8d9
Submitter: Jenkins
Branch: stable/newton

commit d9e584bd4fd95cdb19b89d0edf591c13c10cd8d9
Author: kairat_kushaev <email address hidden>
Date: Thu Nov 3 13:21:05 2016 +0300

    Do not use service catalog for cache client

    Previously cache client used endpoints from service catalog to
    request cache items from node. It works perfectly on devstack
    but it doesn't work when an OpenStack cloud is deployed with HA
    (3/5/7 controllers). We need to use -H option instead.
    This option makes cache management client interact with the local
    endpoint which is more appropriate as image cache is local to
    each node/controller.

    Closes-Bug: #1634486
    (cherry-picked from commit 616060db65ac3f12c0a71fd21f8156c6e993d7ab)

    Change-Id: I355fb36d9af7e06dcac70b8263dc1a2b915ffc8f

[OpenStack Infra (hudson-openstack)]

Change abandoned by Kairat Kushaev (<email address hidden>) on branch: master
Review: https://review.openstack.org/391859