glance-simplestreams-sync does not use new proxy settings

[Dmitrii Shcherbakov]

gss currently uses a shell wrapper to source Juju proxy settings from a certain file.

cat /usr/share/glance-simplestreams-sync/glance-simplestreams-sync.sh
#!/bin/bash
if [ -f /etc/juju-proxy.conf ]; then
    source /etc/juju-proxy.conf
elif [ -f /home/ubuntu/.juju-proxy ]; then
    source /home/ubuntu/.juju-proxy
fi
exec /usr/share/glance-simplestreams-sync/glance-simplestreams-sync.py

However, Juju only saves those files for legacy proxy settings:

Provisioning time
https://github.com/juju/juju/blob/juju-2.6.8/cloudconfig/userdatacfg_unix.go#L280-L286

Proxy updater on model proxy settings changes
https://github.com/juju/juju/blob/juju-2.6.8/worker/proxyupdater/proxyupdater.go#L186-L190
https://github.com/juju/juju/blob/juju-2.6.8/worker/proxyupdater/proxyupdater.go#L150-L157
https://github.com/juju/juju/blob/juju-2.6.8/worker/proxyupdater/proxyupdater.go#L101-L110

Therefore, the charm needs to be reworked to support juju-http-proxy, juju-https-proxy, juju-no-proxy.

Note: proxy settings must only be used for external image downloads, thus only do_sync needs to use the proxy settings.

https://github.com/openstack/charm-glance-simplestreams-sync/blob/77fab981420f49cd399ec89c4db4e9a506d395ad/scripts/glance-simplestreams-sync.py#L242-L296

In other words if /etc/juju-proxy.conf only contained proxy settings, connectivity to Keystone would go through the proxy as well.

For connecting to Keystone the expectation is that it will be reachable via L3, not via proxy (connectivity to via proxy to Keystone API from a charm unit is not something that I have seen so far as we always have direct connectivity to Keystone endpoints).

As no_proxy usage is undesirable at the model level, the fix needs to be targeted at using an http proxy only for image downloads.

[Dmitrii Shcherbakov (dmitriis)]

Subscribed field-medium as glance-simplestreams-sync usage is mentioned in the official guide for using octavia-diskimage-retrofit:

https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-octavia.html#amphora-image

[Dmitrii Shcherbakov (dmitriis)]

The charm should be extended to use env_proxy_settings from charm-helpers to get proxy settings in either format and make glance-simplestreams-sync.py script invocations to use them.

see https://github.com/juju/charm-helpers/pull/248 for more information on handling juju-{http,https,no}-proxy model configs.

[Giuseppe Petralia (peppepetra)]

This is currently blocking the deploy of the charm in 2 Bootstack customers.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/694174
Committed: https://git.openstack.org/cgit/openstack/charm-glance-simplestreams-sync/commit/?id=33b9da2bc5106a33ab0c87fb19056532a94de0dd
Submitter: Zuul
Branch: master

commit 33b9da2bc5106a33ab0c87fb19056532a94de0dd
Author: Alex Balderson <email address hidden>
Date: Wed Nov 13 14:37:45 2019 -0800

    Use current juju proxy config when running sync.

    Adds lines to source current juju proxy environment variables.

    Closes-Bug: #1843486

    Change-Id: I2b461ca6112839effcd7efa3bd9d821f92857584

[Andre Ruiz (andre-ruiz)]

This fix does not seem to work with current charm releases (tested from 30 to 32).

Proxy is configured in juju-*-proxy vars in juju model, but are *not* configured in /etc/environment. If I do set normal http_proxy vars, it works.

All the files sourced in the glance-simplestreams-sync.sh script are empty in the environment I'm deploying. That script is copied to /usr/share and runs from cron. From there, it just cannot use the juju-*-proxy vars anymore, and as I said, all places where it should be are empty.

I created a workaround in which on "install" hook I check if the juju proxy vars are set and write them to /etc/juju-proxy.conf (this is one of the sourced files in the script). It works, fixed the problem. I just don't know if this is the best approach.

The changed code is here: https://code.launchpad.net/~andre-ruiz/+git/charm-glance-simplestreams-sync-proxy-fix

The charm with the applied fix is here: https://jaas.ai/u/andre-ruiz/glance-simplestreams-sync/0

[Andre Ruiz (andre-ruiz)]

Just a note to inform that I updated the charm, latest version is:

cs:~andre-ruiz/glance-simplestreams-sync-1

https://jaas.ai/u/andre-ruiz/glance-simplestreams-sync/1

[Ryan Beisner (1chb1n)]

Reopened based on dup @ https://bugs.launchpad.net/charm-glance-simplestreams-sync/+bug/1883656.

FYI, Additional context in the conversation @ PR https://github.com/juju/juju/pull/11713.

[Ryan Beisner (1chb1n)]

The triage (next steps disposition) from the dup bug:

https://bugs.launchpad.net/charm-glance-simplestreams-sync/+bug/1883656/comments/9

[Vern Hart (vern)]

Merge request in review: https://review.opendev.org/743022/

[Vern Hart (vern)]

For what it's worth, I've been using https://jaas.ai/u/vern/glance-simplestreams-sync/2 while waiting for this merge request to land.

And I also agree with Dmitrii that simplestreams should support --mirror-proxy style arguments so that we can be more targeted with what connections require/use proxies and not have to rely on no_proxy.

[Przemyslaw Hausman (phausman)]

Hi, can we please have this MR merged? We'll need this for the upcoming cloud deployment. Thanks in advance!

[James Page (james-page)]

There are still outstanding questions on the review - hence its not been landed yet. Those reflect the comment in #11

[Vern Hart (vern)]

If we are waiting on changes to sstream-mirror-glance, that comes from python3-simplestreams-openstack, which is built from simplestreams, it should be added to this bug.

I believe the request is to modify sstream-mirror-glance and add the ability to set something like mirror-http-proxy, mirror-https-proxy, mirror-no-proxy for the purpose of connecting to the mirror -- but not for any other connections.

The idea is that communication to the mirror sometimes requires a proxy whereas communication with glance and swift rarely does. In those rare situations, a system-wide proxy could be used. But mirror*proxy settings would allow us to better target what needs a proxy without having to carefully craft a no-proxy variable (which can sometimes be difficult if the subnet where glance and swift are is particularly large and they can get any IP in that subnet).

[Dan Watkins (oddbloke)]

This sounds like a reasonable feature request to me.

[Hybrid512 (walid-moghrabi)]

any news on this ?
It breaks automatic amphora image creation when behind a proxy.

One workaround is to change the juju proxy settings to use the legacy "http_proxy/https_proxy" env variables then redeploy the charm but this makes all other charm deployment failing unless you revert the proxy env variables to new ones.

Best regards

[Andre Ruiz (andre-ruiz)]

I seem to be hitting this with Focal / Ussuri and with Gss charm rev 36. It will just do nothing after installed and set to "run=true". I tested with my modified charm (probably too old now, it was based on gss charm rev 32), did not work. Tested with manually setting proxy vars and calling the script (as per Vern's suggestion) and it also did not work. Will have to investigate more.

[Andre Ruiz (andre-ruiz)]

Just a clarification, the official templates have:

config/openstack_versioned_overlay.yaml: charm: cs:glance-simplestreams-sync-36

While the "stable" is at rev 23.

[Andre Ruiz (andre-ruiz)]

Please ignore my last comment about "stable is at 23", I was mixing numbers with a different problem.

[Nobuto Murata (nobuto)]

Bumping this to ~field-high as it affects all of proxy based customer environments. And we cannot keep using random workarounds in the field.

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Vern Hart <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/743022
Reason: Abandoning to allow someone else to provide a solution.

[Dmitrii Shcherbakov (dmitriis)]

We had a discussion about it during OE team's standup.

While exposing options to control proxy behavior in simplestreams would be preferred, it may take some time to implement this.

A viable alternative seems to be generating a NO_PROXY list from the model context since Juju knows about the Keystone and Swift API units that need to be taken into account for which proxying needs to be avoided.

[Drew Freiberger (afreiberger)]

Can we at least get the sync-images action to pull the env_proxy_settings and append them to the env so that the action does the right thing, while trying to figure out proper settings for the cronjob? Maybe even allowing setting a proxy as an arg to the action. We need a stop-gap that doesn't require logging into the machine to run the sync manually.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/801077

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/801077
Committed: https://opendev.org/openstack/charm-glance-simplestreams-sync/commit/009c8a7b929c9b961c9cf388c122cffe6f6ae41c
Submitter: "Zuul (22348)"
Branch: master

commit 009c8a7b929c9b961c9cf388c122cffe6f6ae41c
Author: Dmitrii Shcherbakov <email address hidden>
Date: Thu Jul 15 20:49:16 2021 +0300

    Dynamically generate proxy settings for image syncs

    sstream-mirror-glance has several endpoints it needs to talk to:

    * Image mirrors - typically, public Internet endpoints;
    * Keystone - typically, a directly reachable endpoint;
    * Glance - typically, a directly reachable endpoint;
    * Object store (Swift) - typically, a directly reachable endpoint but
      sometimes it may be deployed externally and added to the region
      catalog in Keystone (in which case it might be accessible via a proxy
      only).

    While sstream-mirror-glance does not support specifying proxy settings
    for individual directions, since we know all of them based on the
    Keystone catalog, a list of endpoints to add to NO_PROXY environment
    variable can be generated dynamically.

    The complication is that image syncs are periodically done via a cron
    job so a juju-run invocation is needed to retrieve relevant proxy
    settings from model-config at each invocation of the synchronization
    script.

    Additionally, the charm is long-lived so there may be some environments
    that rely on legacy proxy settings. This change accounts for that and
    acts both on juju-prefixed (new) and unprefixed (legacy) proxy settings.

    Whether to use proxy settings for connections to the object store API
    is controlled by a charm option which the script is made to react to.
    Proxy settings are ignored for object store connections by default.

    Closes-Bug: #1843486
    Change-Id: Ib1fc5d2eebf43d5f98bb8ee405a3799802c8b8dc

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/21.04
Review: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/802052

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/openstack/charm-glance-simplestreams-sync/+/802052
Committed: https://opendev.org/openstack/charm-glance-simplestreams-sync/commit/a18cca2ec0d843072c4cc732d6393110ac0124af
Submitter: "Zuul (22348)"
Branch: stable/21.04

commit a18cca2ec0d843072c4cc732d6393110ac0124af
Author: Dmitrii Shcherbakov <email address hidden>
Date: Thu Jul 15 20:49:16 2021 +0300

    Dynamically generate proxy settings for image syncs

    sstream-mirror-glance has several endpoints it needs to talk to:

    * Image mirrors - typically, public Internet endpoints;
    * Keystone - typically, a directly reachable endpoint;
    * Glance - typically, a directly reachable endpoint;
    * Object store (Swift) - typically, a directly reachable endpoint but
      sometimes it may be deployed externally and added to the region
      catalog in Keystone (in which case it might be accessible via a proxy
      only).

    While sstream-mirror-glance does not support specifying proxy settings
    for individual directions, since we know all of them based on the
    Keystone catalog, a list of endpoints to add to NO_PROXY environment
    variable can be generated dynamically.

    The complication is that image syncs are periodically done via a cron
    job so a juju-run invocation is needed to retrieve relevant proxy
    settings from model-config at each invocation of the synchronization
    script.

    Additionally, the charm is long-lived so there may be some environments
    that rely on legacy proxy settings. This change accounts for that and
    acts both on juju-prefixed (new) and unprefixed (legacy) proxy settings.

    Whether to use proxy settings for connections to the object store API
    is controlled by a charm option which the script is made to react to.
    Proxy settings are ignored for object store connections by default.

    Closes-Bug: #1843486
    Change-Id: Ib1fc5d2eebf43d5f98bb8ee405a3799802c8b8dc
    (cherry picked from commit 009c8a7b929c9b961c9cf388c122cffe6f6ae41c)

[Nobuto Murata (nobuto)]

Unsubscribing ~field-high since the stable backport of the charm has been completed and the other task attached to this bug as simplestreams itself is not urgent at this moment.