g-s-s action sync-images silently fails

[Przemyslaw Hausman]

g-s-s uses OpenStack hardcoded 'public' endpoint for connecting to keystone, even though the binding for 'identity-service' is explicitly configured to use network space relevant for 'internal' OpenStack endpoint.

As a result, g-s-s is not able to connect to keystone and the action `sync-images` silently fails like this:

$ juju run-action glance-simplestreams-sync/0 sync-images --wait
unit-glance-simplestreams-sync-0:
  UnitId: glance-simplestreams-sync/0
  id: "48"
  results: {}
  status: completed
  timing:
    completed: 2021-10-06 11:23:35 +0000 UTC
    enqueued: 2021-10-06 11:23:34 +0000 UTC
    started: 2021-10-06 11:23:34 +0000 UTC

1. Keystone endpoints

$ openstack endpoint list --format value -c 'Service Name' -c Interface -c URL | grep keystone
keystone public https://keystone.example.com:5000/v3
keystone internal https://keystone-internal.example.com:5000/v3
keystone admin https://keystone-internal.example.com:35357/v3

2. g-s-s has 'identity-service' binding configured to 'internal-space'

$ juju status --format yaml glance-simplestreams-sync/0
model:
  name: openstack
  [...]
applications:
  glance-simplestreams-sync:
    charm: cs:glance-simplestreams-sync-43
    [...]
    endpoint-bindings:
      "": oam-space
      certificates: oam-space
      identity-service: internal-space
      image-modifier: oam-space
      nrpe-external-master: oam-space
      simplestreams-image-service: oam-space

3. g-s-s and keystone relation data. Note that keystone-internal.example.com:5000 should be used for further communication with keystone -- so 'internal_host' and 'internal_port' relation data should be considered.

$ juju run --unit glance-simplestreams-sync/0 'relation-get -r identity-service:111 - keystone/0'
admin_domain_id: 6088d6b81a594aae991ed08b0469cb2f
admin_project_id: 2c1c659bcbf34133b81d9f25e0d98300
admin_user_id: 933145c81baa4429a4c332d46c89550e
api_version: "3"
auth_host: keystone-internal.example.com
auth_port: "35357"
auth_protocol: https
egress-subnets: 10.254.0.43/32
ingress-address: 10.254.0.43
internal_host: keystone-internal.example.com
internal_port: "5000"
internal_protocol: https
private-address: 10.254.0.43
service_domain: service_domain
service_domain_id: ff9a17cfed754a778a56ae83c0c4f0d6
service_host: keystone.example.com
service_password: KskTktBCgXCb8PtzC853JxhBkKB3jNnK8VJxkRcg4j6x5cY6Z3S8nfYR4BcRrH5x
service_port: "5000"
service_protocol: https
service_tenant: services
service_tenant_id: a4286e2f37234280813c4a347960fa42
service_username: image-stream

4. Relation data rendered to templates/identity.yaml for g-s-s unit is missing 'internal_host' and 'internal_port'

# cat templates/identity.yaml
  1
  2 api_version: {{ api_version }}
  3 auth_host: {{ auth_host }}
  4 auth_port: {{ auth_port }}
  5 auth_protocol: {{ auth_protocol }}
  6 service_host: {{ service_host }}
  7 service_port: {{ service_port }}
  8 service_protocol: {{ service_protocol }}
  9 admin_tenant_id: {{ admin_tenant_id }}
 10 admin_tenant_name: {{ admin_tenant_name }}
 11 admin_user: {{ admin_user }}
 12 admin_password: {{ admin_password }}
 13 {% if ssl_ca -%}
 14 ssl_ca: |
 15 {{ ssl_ca | indent( width=2, indentfirst=True) }}
 16 {% endif -%}
 17
 18 {% if api_version == '3' -%}
 19 admin_domain_name: {{ admin_domain_name }}
 20 {% endif -%}
 21 unit_name: {{ unit_name }}

5. g-s-s uses hardcoded 'service_host' and 'service_port' values for connecting to keystone, see lines 209, 210 of /usr/share/glance-simplestreams-sync/glance_simplestreams_sync.py

205 def set_openstack_env(id_conf, charm_conf):
206 version = 'v3' if str(id_conf['api_version']).startswith('3') else 'v2.0'
207 auth_url = ("{protocol}://{host}:{port}/{version}"
208 .format(protocol=id_conf['service_protocol'],
209 host=id_conf['service_host'],
210 port=id_conf['service_port'],
211 version=version))

6. g-s-s log confirms that the charm is trying to access public endpoint instead of internal endpoint:

INFO * 10-06 07:08:01 [PID:20347] * root * glance-simplestreams-sync started.
DEBUG * 10-06 07:08:01 [PID:20347] * keystoneclient.auth.identity.v3.base * Making authentication request to https://keystone.example.com:5000/v3/auth/tokens
DEBUG * 10-06 07:08:01 [PID:20347] * urllib3.connectionpool * Starting new HTTPS connection (1): keystone.example.com:5000

EXPECTED BEHAVIOUR

1. g-s-s should be connecting to keystone using the space configured for the binding 'identity-service'.
2. In case of a failure, the output of `sync-images` actions should provide meaningful imformation to make troubleshooting easier.

[Billy Olsen (billy-olsen)]

Charms in the openstack collection typically have a `use-internal-endpoints` option for influencing the endpoints which are used. This was implemented in https://bugs.launchpad.net/charm-glance-simplestreams-sync/+bug/1896438 but not yet released into a stable charm. Marking as duplicate