No way to tell glance-simplestreams-sync to use internal endpoints

[Vern Hart]

I am deploying an environment where the public endpoints are only accessible from outside the deployment. This means all openstack services need to use internal endpoints.

The glance-simplestreams-sync charm, which uses the simplestreams snap, does not have a way of requesting internal endpoints be used.

[Nikolay Vinogradov (nikolay.vinogradov)]

I'm seeing the same on the current deployment. I have to add glance-simplestreams-sync to public API subnet which doesn't make sense.

[Gábor Mészáros (gabor.meszaros)]

Same here.

[Gábor Mészáros (gabor.meszaros)]

I'm also having the same issue in my running deployment.
My deployment is not even erroring out when trying to access the public endpoint of keystone, it just retries infinitely.
No proxy env, just public subnet is not available on the simplestreams container (neither switched, nor routed).
glance-simplestreams-sync.log is full with:

DEBUG * 09-21 16:08:01 [PID:47111] * keystoneclient.auth.identity.v3.base * Making authentication request to https://keystone.public.domain:5000/v3/auth/tokens
DEBUG * 09-21 16:08:01 [PID:47111] * urllib3.connectionpool * Starting new HTTPS connection (1): keystone.public.domain
INFO * 09-21 16:09:01 [PID:47125] * root * glance-simplestreams-sync started.
INFO * 09-21 16:09:01 [PID:47125] * root * /var/run/glance-simplestreams-sync.pid is locked, exiting
INFO * 09-21 16:10:02 [PID:47141] * root * glance-simplestreams-sync started.
INFO * 09-21 16:10:02 [PID:47141] * root * /var/run/glance-simplestreams-sync.pid is locked, exiting
INFO * 09-21 16:11:01 [PID:47375] * root * glance-simplestreams-sync started.

[Vern Hart (vern)]

In the identity relation to glance-simplestreams-sync, keystone provides some auth_* variables and some service_* variables. In my deployment the service_host has the public keystone hostname and the auth_host has the internal/admin hostname. So to push this forward a bit I modified /var/run/glance_simplestreams_sync.py to use the auth_* variables in the set_openstack_env function.

And to get it to use the swift (radosgw) internal endpoint, I put the public hostname in /etc/hosts with the internal IP. This allows me to connect but now I'm getting 401 Unauthorized from radosgw. This may or may not be related.

Obviously this is not a fix. Just a work-around.

[Nobuto Murata (nobuto)]

Subscribing ~field-high. Having "use-internal-endpoint" option or equivalent is one of the minimal expected feature set in OpenStack charms/services. Otherwise we have to put every single node/service onto an external/public network.

[Paride Legovini (paride)]

Hi,

Can anything be done on the simplestreams side to overcome the problem described by this bug? IIUC this is a limitation of the glance-simplestreams-sync charm and not of simplestreams itself, which always allows the mirror_url to be specified, but admittedly I know very little about charms, so do not hesitate to fixme. Thanks!

[Jeff Hillman (jhillman)]

I am seeing this issue as well. My case is slightly different though. I didn't have an interface on the public network, but the public network was routable. However, even though I could ping and telnet to the public IP of keystone, I couldn't properly talk to it.

In either case, if I could have forced GSSS to use just the keystone-int then there would have been no issue to begin with.

[Camille Rodriguez (camille.rodriguez)]

I'm seeing this as well. My workaround was to go in the unit and edit /etc/glance-simplestreams-sync/identity.yaml and make sure that the keystone endpoint would be the internal endpoint (keystone-internal.customer.cloud.com) instead of the public endpoint.

[Billy Olsen (billy-olsen)]

I understand the pain this causes in deployments, but as a feature request to add this support to the charm it is not eligible for field sla. Will come up with a plan to work on this.

[Nobuto Murata (nobuto)]

@Billy, I personally disagree with the statement. In any case, we need to see the clear list of minimum expected functionality of OpenStack charms so we can avoid ambiguity.

For example:
- High-availability (with vip or DNS HA)
- TLS encryption for endpoints (with vault or ssl_* options)

and "use-internal-endpoint" should be one of them since if one charm is missing it then it will ruin the idea of having "use-internal-endpoint" by other charms and ruin the network segregation and design.

> Having "use-internal-endpoint" option or equivalent is one of the minimal expected feature set in OpenStack charms/services. Otherwise we have to put every single node/service onto an external/public network.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/804924

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/804924
Committed: https://opendev.org/openstack/charm-glance-simplestreams-sync/commit/6733e22006750fa0a0dc405278b0446fd4aecac4
Submitter: "Zuul (22348)"
Branch: master

commit 6733e22006750fa0a0dc405278b0446fd4aecac4
Author: Billy Olsen <email address hidden>
Date: Tue Aug 17 09:12:02 2021 -0700

    Add use-internal-endpoints as a new config option

    Enable the glance-simplestreams-sync charm to be configured to only
    use internal endpoints, removing the requirement that the charm
    access APIs via the public network.

    This introduces a new option use-internal-endpoints, which is in-line
    with other openstack charms which can be configured to use the internal
    endpoints rather than public endpoints.

    This option will only be valid for Keystone v3 API versions, which
    should not be problematic as the v2 API has been deprecated for quite
    some time now.

    Closes-Bug: #1896438
    Change-Id: I6d83a5469c34c2add3c1bc4ac5052be6d51ed343

[Przemyslaw Hausman (phausman)]

The above commit (https://bugs.launchpad.net/charm-glance-simplestreams-sync/+bug/1896438/comments/12) only partially solves the problem. I understand that simplestreams need to be patched too.

@paride, it looks to me that simplestreams does not take into account OS_ENDPOINT_TYPE environment variable. And the above commit to glance-simplestreams-sync charm in fact sets this environment variable.

Since simplestreams does not process OS_ENDPOINT_TYPE environment variable, it creates swift and glance clients, that by default try to connect with the public OpenStack endpoints. In the environments where public endpoints are not accessible (e.g. there's no routing to the VLAN where OpenStack public endpoints are exposed), simplestreams will get stuck while trying to connect to swift and glance public endpoints. I believe this should be changed.

If the OS_ENDPOINT_TYPE environment variable is set, simplestreams should populate the os_options['endpoint_type'] parameter with the value of OS_ENDPOINT_TYPE (e.g. 'internal') while creating swift and glance clients in functions get_swiftclient() [1] and get_glanceclient() [2].

1. https://git.launchpad.net/simplestreams/tree/simplestreams/objectstores/swift.py#n27
2. https://git.launchpad.net/simplestreams/tree/simplestreams/mirrors/glance.py#n34

Also both get_swiftclient() and get_glanceclient() functions should not reuse session (if it already exists) because this session does not have os_options populated.

To make the above work, two other places need to be patched.

openstack.load_keystone_creds() [3] does not read environment variable OS_ENDPOINT_TYPE set by glance-simplestreams-sync charm in [4]. This needs to be patched by adding 'OS_ENDPOINT_TYPE' to OS_ENV_VARS [5].

3. https://git.launchpad.net/simplestreams/tree/simplestreams/openstack.py#n73
4. https://opendev.org/openstack/charm-glance-simplestreams-sync/src/commit/6733e22006750fa0a0dc405278b0446fd4aecac4/files/glance_simplestreams_sync.py#L239
5. https://git.launchpad.net/simplestreams/tree/simplestreams/openstack.py#n33

openstack.get_service_conn_info() [6] does not return 'endpoint_type' key in the 'info' dict. This needs to be patched, e.g. by adding 'endpoint_type': kwargs.get('endpoint_type')} to info dict.

6. https://git.launchpad.net/simplestreams/tree/simplestreams/openstack.py#n194

Hope it makes sense,
Przemyslaw

[Przemyslaw Hausman (phausman)]

I used the -next version of g-s-s with the commit from #12 applied and also applied a workaround by @vern from #4. The workaround worked for me.

This hack is a bit risky in production environments so I'm attaching field-high, as we are seeing this issue on multiple deployments.