snap-http-proxy and snap-https-proxy not honored in juju 2.4.1

[Drew Freiberger]

In an environment where I must proxy all APT and snapstore traffic through my MAAS node (10.216.2.0/16), I've set up the juju model-config on juju 2.4.1 to set snap-http-proxy and snap-https-proxy to http://10.216.2.0/16, and yet when I install a snap-based charm into a new LXD container with "juju add-unit etcd --to lxd:<machine number>", I get a failed hook on install:

Unit error log snippet: https://pastebin.ubuntu.com/p/YTMp2HQkgj/

Model-config: https://pastebin.ubuntu.com/p/GrfrCphjmY/

However, if I set http-proxy and https-proxy in my model, restart snapd on the affected unit and then re-run the install hook succeeds.

This either is my misunderstanding of what snap-http-proxy is supposed to provide (assumed it was access to the store) or the functionality is not working as intended because the charm is overriding it with it's own snap-proxy settings that are blank.

Should this juju model snap-http-proxy config override the charm's settings of such?

[Drew Freiberger (afreiberger)]

I'm guessing that this charm being layer-etcd, that it's using layer-apt which would possibly be intermingling settings from the charm configs and juju model.

[Drew Freiberger (afreiberger)]

Marking field-critical as etcd on proxy-bound sites under juju 2.4.1 does not appear to be able to either install, or cluster.

Under 2.3.8, it clusters fine with http-proxy and https-proxy set. This behavior seems to be 2.4.x and proxy settings related.

Other references:

https://github.com/juju-solutions/layer-etcd/issues/122

[Tim Penhey (thumper)]

To exclude the charm getting in the way, we can test this by just shelling into the machines.

The snap-http-proxy values are set by the machine agent for the entire machine.

First check that the proxy values are set for the core snap:

`sudo snap get core proxy`

I'd then check just installing the snap manually:

`snap install --channel=3.1/stable --classic etcd`

And check on what that does. If this works, then it is a charm issue.

It is always possible that the charm is calling into `snap set core` and is overriding the values that Juju has set. It would have needed to do this to handle snap proxies before Juju handled them. Could the snap layer be doing this?

[Tim Penhey (thumper)]

The base snap layer doesn't take into account existing snap proxies set.

https://git.launchpad.net/layer-snap/tree/reactive/snap.py line 150.

It looks for a file it has set rather than checking what the core snap has set.

It will also just override whatever has been set for snap proxy values with what it determines from the environment. See line 129 of the same file.

Awaiting confirmation before marking invalid for Juju.

[Xav Paice (xavpaice)]

Indeed, running:

juju config etcd snap_proxy=http://my.proxy:3128

Does seem to allow the snap to install for etcd at least.

I still can't get it clustered, and https://github.com/juju-solutions/layer-etcd/issues/122 indicates that this is because the proxy vars aren't working properly.

[Stuart Bishop (stub)]

What should be the correct behavior for the snap layer?

I can't find the snap-http-proxy model config documented in the Juju docs.

https://bugs.launchpad.net/snapcraft/+bug/1533899 is still open on the snapd side, so web proxy support on that side still seems to be via hacks.

[Tim Penhey (thumper)]

After discussions with the snapd team, it was decided that Juju wouldn't write a file to disk like it did with the http proxies, but instead call snap directly to set the values.

`sudo snap get core proxy` will give you the proxies used for snapd.

```
$ sudo snap get core proxy
Key Value
proxy.http
proxy.https
proxy.store
```

If you call with just a single value, `sudo snap get core proxy.http` you will get the value back.

Juju doesn't expose the snap proxy variables to the charm because they shouldn't need to know.

I don't know why the snapd team hasn't closed bug 1533899.

Older versions of Juju don't write the values out. I'd recommend that the snap layer changes to using

sudo snap set core proxy.http=foo proxy.https=bar

to set the proxy values, rather than writing a file out.

[Stuart Bishop (stub)]

Per lp:1533899, further work on the snap layer needs to be deferred until the snap core settings approach actually works with non-Core snapd deployments.

[Stuart Bishop (stub)]

Per original bug comment, the model config settings will likely not work with the current release of snapd (2.34/2.35). The snap_proxy charm config option added by the snap layer will need to remain in use.

[Drew Freiberger (afreiberger)]

It appears that this PR is going to be the solution: https://github.com/snapcore/snapd/pull/5699

The juju model-config IS setting the proper variables, snapd just isn't acting on the settings. Thanks for chasing that down, @thumper and @stub.

[Stuart Bishop (stub)]

Once the snapd patch is released (hopefully as far as trusty), we need to ensure that the snap layer does not override the model config settings. Also, the snap layer proxy config item needs to be documented as deprecated.

[Chris Gregan (cgregan)]

This issue has stalled for too long. Is it actually resolved? I thought it was, but seems to be back as active critical.

[Stuart Bishop (stub)]

I've just documented the snap layer proxy settings as deprecated, and don't think that the snap-layer settings will interfere with the new juju model config (which should now be working for all supported Ubuntu series).