Does not use juju_http_proxy for cloud-images.ubuntu.com

[Vern Hart]

I'm using glance-simplestreams-sync-30 (commit f45f2d55eda48f577d9d8d69abd533d9fb763b39) and juju 2.7.6, though that's probably not relevant.

When running glance-simplestreams-sync behind a proxy, it's failing to sync images.

In /var/log/glance-simplestreams-sync.log I see

    Starting new HTTP connection (1): cloud-images.ubuntu.com

Followed by a traceback ending in:

    ConnectionError: HTTPConnectionPool(host='cloud-images.ubuntu.com', port=80): Max retries exceeded with url: /releases/streams/v1/index.sjson (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f771e9bc710>: Failed to establish a new connection: [Errno 101] Network is unreachable',))

In my model's config, I have juju proxy variables set but not the legacy proxy variables:

    $ juju model-config | grep proxy
    apt-ftp-proxy default ""
    apt-http-proxy controller http://myproxy:8000
    apt-https-proxy controller http://myproxy:8000
    apt-no-proxy default ""
    ftp-proxy default ""
    http-proxy default ""
    https-proxy default ""
    juju-ftp-proxy default ""
    juju-http-proxy controller http://myproxy:8000
    juju-https-proxy controller http://myproxy:8000
    juju-no-proxy controller 127.0.0.1,localhost,::1,.myopenstack-domain.com,10.0.0.0/8
    no-proxy default 127.0.0.1,localhost,::1
    proxy-ssh default false
    snap-http-proxy controller http://myproxy:8000
    snap-https-proxy controller http://myproxy:8000
    snap-store-proxy default ""
    snap-store-proxy-url default ""

A previously submitted fix adds two lines that read in /etc/profile.d/juju-proxy.sh if it exists. In my deployment, that file simply contains:

    [ -f "/etc/juju-proxy.conf" ] && . "/etc/juju-proxy.conf"

And /etc/juju-proxy.conf (which was already being read in prior to the fix) is empty.

In the charm hooks environment I have:

    JUJU_CHARM_HTTPS_PROXY=http://myproxy:8000
    JUJU_CHARM_HTTP_PROXY=http://myproxy:8000
    JUJU_CHARM_NO_PROXY=127.0.0.1,localhost,::1,.myopenstack-domain.com,10.0.0.0/8

I was able to temporarily work around this by connecting to the unit and running:

    export http_proxy=http://myproxy:8000
    export no_proxy=.myopenstack-domain.com
    sudo -E /usr/share/glance-simplestreams-sync/glance-simplestreams-sync.sh

(I had to run that last one several times to catch a free moment between the once-a-minute cron job updates.)

I would prefer not to use http_proxy in my model config because that would cover all system connections on all machines and my subnets are too big to fit in no_proxy (which doesn't support cidr like juju_no_proxy).

Can we use the juju proxy settings for the cloud-images.ubuntu.com but not for the connections to keystone?

[Dmitrii Shcherbakov (dmitriis)]

Added Juju since /etc/juju-proxy.conf is only written to when legacy proxy settings are set.

[Dmitrii Shcherbakov (dmitriis)]

Suggested a Juju-side change here: https://github.com/juju/juju/pull/11713

[Ian Booth (wallyworld)]

If I recall correctly, the juju proxy settings were specifically designed not to be used to interfere with the system wide proxy settings. This was because issues arose when juju needed specific values set that then messed up the non-juju networking requirements of the host.

The idea is that charm and other juju artefacts explicitly can query the various juju proxy values and do the right thing. If a specific charm needs to set system proxy values from the juju ones, it can do so. But, by design, this is not the default behaviour. I haven't got specific examples I can relate as the the issues that were encountered, but it was a mess.

[Vern Hart (vern)]

I tend to agree that in the situation where we're not using the legacy proxy settings (http_proxy, etc.) it should be up to the charm on how to apply proxy settings.

I believe that's why we have the JUJU_CHARM_*_PROXY environment variables during hook execution.

In the case of glance-simplestreams-sync, it's a cron job that's executed outside the hook environment so the proxy variables would need to be saved to /etc/glance-simplestreams-sync.conf during the install and/or config-changed hooks.

Is it reasonable for the glance-simplestreams-sync charm to assume proxy (if supplied) should be used for cloud-images.ubuntu.com but not for the openstack connections? Or should it always apply proxy and no_proxy info on every connection? It seems like in the majority of situations, it'd be safe to assume.

[Dmitrii Shcherbakov (dmitriis)]

> If I recall correctly, the juju proxy settings were specifically designed not to be used to interfere with the system wide proxy settings.

Yes, that is the current implementation and intent of my patch as well. I am not suggesting we automatically inject these settings into the environment of every process or service - this is bad for sure.

> I haven't got specific examples I can relate as the the issues that were encountered

There were 2 issues originally:

1) auto-injection of environment variables into the default service and shell environments.
2) auto-injection of environment variables that modify http client behavior into the *hook environment*;

(1) got addressed by writing to files not picked up by any tooling automatically (just files under /etc/):

/etc/juju-proxy-systemd.conf # (this file needs to be symlinked manually to be used per the comment here https://github.com/juju/proxy/blob/master/proxy.go#L108-L110)
/etc/juju-proxy.conf # this file needs to be sourced manually to be used

(2) got addressed by adding proxy settings with a "juju-" prefix and JUJU_CHARM_* environment variables in hook environments.

However, (1) is currently only done for legacy proxy settings, not the new proxy settings.

a) on initial startup via the cloud-init-userdata;
https://git.io/JfdGL

b) by proxy-updater worker on any changes to model config
https://git.io/JfdsA
https://git.io/Jfdsp (note: the comment here is old, settings are not written to /etc/systemd/system.conf.d or /etc/systemd/user.conf.d nowadays for legacy proxy settings)
https://git.io/JfdGj (likewise, only done for legacy proxy settings).

My patch was an attempt to have the above behavior for new proxy settings as well since it doesn't affect processes globally (I only addressed (a) but missed (b) in the current version of the PR).

The reason is the lack of a way to track proxy setting updates:

* there is no hook for that currently: we would have to track updates at the charm library level on every hook execution;
* proxy worker updates are asynchronous to the hook execution - there will always be a lag between when they get updated and hook execution (update-status executes every 5 minutes by default).

[Dmitrii Shcherbakov (dmitriis)]

Having /etc/juju-proxy.conf written to would only give us a way to write wrappers that would allow sourcing proxy variables for:

* cron jobs or timers;
  * which is what we have with glance-simplestreams-sync;
* software that periodically executes subprocesses (through wrappers that would load /etc/juju-proxy.conf).

For services that require a manual restart or reload to see a change to settings in /etc/juju-proxy.conf we would still need a hook to fire to make a charm do a restart/reload of the necessary services.

In summary:

* for the purposes of glance-simplestreams-sync, the easiest would be to have /etc/juju-proxy.conf written to via cloudinit-userdata/proxy updater since we use a cron job;
* writing proxy settings to a file such as /etc/glance-simplestreams-sync.conf would not solve the problem with asynchronous updates to proxy settings - we need hooks to fire for that.

[Vern Hart (vern)]

The proposed change puts juju proxy settings into /etc/juju-proxy.conf but it should be noted that /etc/profile.d/juju-proxy.sh (a file sourced for all interactive shells) contains:

    # Added by juju
    [ -f "/etc/juju-proxy.conf" ] && . "/etc/juju-proxy.conf"

Which means putting anything in /etc/juju-proxy.conf will affect all users on the system. If we want to affect all users, we should use the legacy proxy variables, no?

[Dmitrii Shcherbakov (dmitriis)]

Vern,

I have not modified that part in my PR, however, I think writing to /etc/profile.d/juju-proxy.sh should only be done for the legacy proxy settings as you say.

https://github.com/juju/juju/pull/11713/files#diff-2e72a75bbc42f70552be8a9092cc7f8fL322-L323

I suppose it was not a problem with the new proxy settings since /etc/juju-proxy.conf was never written to and there is a check for the existence of that file in the resulting juju-proxy.sh.

[Dmitrii Shcherbakov (dmitriis)]

Synced with jamespage on that and he suggested an alternative approach, since we already interact with juju via `juju-run` from the script that gets executed via a cron job, we could steal proxy settings from the `juju-run` environment.

https://opendev.org/openstack/charm-glance-simplestreams-sync/src/branch/stable/20.05/files/glance-simplestreams-sync.py#L373

I checked that proxy settings are available from the unit context in a juju-run invocation:

$ sudo juju-run <unit-name> env | grep _PROXY
JUJU_CHARM_FTP_PROXY=
JUJU_CHARM_HTTPS_PROXY=
JUJU_CHARM_HTTP_PROXY=
JUJU_CHARM_NO_PROXY=10.0.0.0/8,192.168.0.0/16,172.16.0.0/12

This allows us to query proxy settings dynamically and have the latest in-agent version at the time of the sync script invocation - this means we don't have to wait until proxy settings change notifications appear in Juju to fix this or do a subjective change to juju to write to /etc/juju-proxy.conf.

[Andre Ruiz (andre-ruiz)]

https://bugs.launchpad.net/charm-glance-simplestreams-sync/+bug/1843486

This seems a dup, I just commented on that one.

[Vern Hart (vern)]

The other bug (LP:1843486) is marked Fixed Released so we might as well keep this one.

I think the idea Dmitrii and James page were kicking around is to modify the python script to make a juju-run call to get the JUJU_CHARM* proxy variables from the environment. Do we need to modify those variables or just export them into the environment?