[2.3.1] snapd proxy settings are not set on deployed machines or containers

[Dmitrii Shcherbakov]

Currently Juju doesn't propagate proxy settings for snapd from model-config to either hosts or LXD containers.

It is quite easy to do so though:

sudo mkdir -p /etc/systemd/system/snapd.service.d/
sudo cat <<EOF | sudo tee -a /etc/systemd/system/snapd.service.d/proxy.conf
[Service]
Environment=http_proxy=$http_proxy
Environment=https_proxy=$http_proxy
EOF
sudo systemctl daemon-reload
sudo systemctl restart snapd # <- mandatory

https://bugs.launchpad.net/snapcraft/+bug/1533899
https://bugs.launchpad.net/snapcraft/+bug/1533899/comments/1

Seems to me snapd reacts to environment variables just because of how golang runtime handles that:

https://github.com/golang/go/blob/release-branch.go1.9/src/net/http/transport.go#L250-L266

snapd *MUST* be restarted though after proxy.conf is created as naturally there is no sane way to "reload" environment variables and make a process react to that without some effort on the process side itself.

Note that this is not about using core snap to set proxy settings - initially you have **no snaps at all** which means you cannot use a core snap to set some settings. `snap install core` will obviously fail at this point so this is not interesting https://github.com/snapcore/snapd/pull/3594

/etc/environment isn't good for this too because no_proxy variable has to be populated with a list of ip addresses and hostnames which becomes quite large if you use a /24 or a /16 network (a /16 would even cause your programs to run out of stack memory due to an 8MiB RLIMIT_STACK default).

pam_env has certain limits which is why we should not use this file:
https://github.com/linux-pam/linux-pam/blob/master/modules/pam_env/pam_env.c#L541
  /* No unexpanded variable can be bigger than BUF_SIZE */
  char type, tmpval[BUF_SIZE];

How private snap stores would fit into that should be evaluated but my primary concern is proxy settings for this bug.

[Dmitrii Shcherbakov (dmitriis)]

Also adding snapd project here because if there is no core snap at the beginning then we cannot set anything, including proxy settings or snap store settings (proxy.store):

https://github.com/snapcore/snapd/blob/release/2.30/corecfg/proxy.go#L78-L96

"No core snap" condition can be easily validated by launching a container:

lxc launch ubuntu:xenial snapdtest ; lxc exec snapdtest snap list
Creating snapdtest
Starting snapdtest
No snaps are installed yet. Try "snap install hello-world".

[Michael Vogt (mvo)]

So for the snapd side of this bug:
1. snapd in edge (2.30+) supports setting core settings even if no core snap is installed.
2. proxy settings are only considered on core currently via /etc/environment and require a restart
3. We are writing /etc/environment on core devices but do no length checks (i.e. we do not check the 8MiB limit)

So (1) is taken care of but (2) and (3) still needs work.

[Michael Vogt (mvo)]

I created https://github.com/snapcore/snapd/pull/4424 to deal with (3), i.e. ensure we do not write a /etc/environment that is too long.

[Dmitrii Shcherbakov (dmitriis)]

Michael,

I am not sure writing to /etc/environment is the right thing to do in all cases.

After working with isolated environments for some time I am more inclined to giving as much granularity as possible because there are distinct use-cases:

1) proxy servers for package-management purposes or image downloads only (snapd, apt, lxd, MAAS, Juju tools etc.);
2) application-specific proxy server configuration (either explicit options in software or drop-in unit files);
3) global proxy server settings for all applications.

OpenStack is a good example because it has http services and can easily hit no_proxy stack memory scalability issues.

- most of the OpenStack services expose an HTTP API;
- some OpenStack services talk to each other by utilizing those HTTP APIs (e.g. Keystone);
- if you use a proxy server to install packages or get cloud images for MAAS or LXD it doesn't mean http clients of deployed services should use http(s)_proxy and no_proxy variables. However, they will try to if /etc/environment will be used.

I think there should be two sets of core options:

* snap-global proxy settings - all snaps should get those variables, not the whole system because the setting is quite significant for a classic system;
  - this might get messy because classic snaps on a classic system do not get a separate mount namespace AFAIR, and hence would read /etc/environment directly from rootfs which we wouldn't be able to control separately;
* snapd-specific proxy settings (searching for snaps, snap downloads and other daemon-specific operations only).
  - this would potentially affect an ability to access the enterprise snap store. I would assume that if you specify this option and a snap store one, accessing a local snap store would go through that proxy server (it's a strange but valid scenario).

Maybe snap-specific options could also be implemented for cases where a snap author has not provided their own - haven't looked deep enough but if snapcraft.yaml allows to declare daemons then creating drop-in directories automatically might be possible. If an application runtime or a library parses those options then it would allow granular control.

I think that using Environment or EnvironmentFile directives on a per-service basis is something to utilize here for snapd-related operations (the config I used in the original description):

https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Description
"Along with a unit file foo.service, a "drop-in" directory foo.service.d/ may exist."
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment
https://www.freedesktop.org/software/systemd/man/systemd.exec.html#EnvironmentFile=

What I am trying to avoid is using a global setting for everything that would require passing a huge blob for no_proxy in case of large subnets. Deployed services would normally need to talk to each other in an isolated environment which would not require any proxy configuration at all so we need to separate out package delivery options from the rest of the configuration.

To sum up: using proxy settings *only* for artifact retrieval (packages and images downloads) is the main field use-case I ...

[John A Meinel (jameinel)]

In practice, is it common to use the same proxy for APT and Snap? Certainly I only have an APT proxy set up on my local network (for caching between multiple machine downloads), but I suppose if I wanted to start caching Snap downloads then I'd probably adapt my proxy rather than running a second one.

Juju could either grow an "archive proxy" which would then be used for:
 1) apt
 2) snap
 3) lxd
 4) juju itself hitting streams.canonical.com for agent binaries?

For some of these we currently also allow you to instead change the target of image-stream/agent-stream/apt-mirror.

It seems like we could have a meta setting, but it may be better to just add individual settings for all the other ones.

[Dmitrii Shcherbakov (dmitriis)]

It's usually one proxy but could be that a local snap store would be used
which would then use that proxy itself. The store URL would be a separate
setting like apt mirror though.

MAAS provides a squid proxy so the flow would be:

Machine -> MAAS proxy -> local snap store -> corporate proxy

Machine -> MAAS proxy -> corporate proxy -> public store

On Mon, Mar 26, 2018, 03:41 John A Meinel <email address hidden> wrote:

> In practice, is it common to use the same proxy for APT and Snap?
> Certainly I only have an APT proxy set up on my local network (for
> caching between multiple machine downloads), but I suppose if I wanted
> to start caching Snap downloads then I'd probably adapt my proxy rather
> than running a second one.
>
> Juju could either grow an "archive proxy" which would then be used for:
> 1) apt
> 2) snap
> 3) lxd
> 4) juju itself hitting streams.canonical.com for agent binaries?
>
> For some of these we currently also allow you to instead change the
> target of image-stream/agent-stream/apt-mirror.
>
> It seems like we could have a meta setting, but it may be better to just
> add individual settings for all the other ones.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1737332
>
> Title:
> [2.3.1] snapd proxy settings are not set on deployed machines or
> containers
>
> Status in juju:
> Triaged
> Status in snapd:
> Triaged
>
> Bug description:
> Currently Juju doesn't propagate proxy settings for snapd from model-
> config to either hosts or LXD containers.
>
> It is quite easy to do so though:
>
> sudo mkdir -p /etc/systemd/system/snapd.service.d/
> sudo cat <<EOF | sudo tee -a
> /etc/systemd/system/snapd.service.d/proxy.conf
> [Service]
> Environment=http_proxy=$http_proxy
> Environment=https_proxy=$http_proxy
> EOF
> sudo systemctl daemon-reload
> sudo systemctl restart snapd # <- mandatory
>
> https://bugs.launchpad.net/snapcraft/+bug/1533899
> https://bugs.launchpad.net/snapcraft/+bug/1533899/comments/1
>
>
> Seems to me snapd reacts to environment variables just because of how
> golang runtime handles that:
>
> https://github.com/golang/go/blob/release-
> branch.go1.9/src/net/http/transport.go#L250-L266
>
> snapd *MUST* be restarted though after proxy.conf is created as
> naturally there is no sane way to "reload" environment variables and
> make a process react to that without some effort on the process side
> itself.
>
> Note that this is not about using core snap to set proxy settings -
> initially you have **no snaps at all** which means you cannot use a
> core snap to set some settings. `snap install core` will obviously
> fail at this point so this is not interesting
> https://github.com/snapcore/snapd/pull/3594
>
> /etc/environment isn't good for this too because no_proxy variable has
> to be populated with a list of ip addresses and hostnames which
> becomes quite large if you use a /24 or a /16 network (a /16 would
> even cause your programs to run out of stack memory due to an 8MiB
> RLIMIT_STACK default).
>
> pam_env has certain limits which is why we sho...

[Dmitrii Shcherbakov (dmitriis)]

John,

> It seems like we could have a meta setting, but it may be better to just add individual settings for all the other ones.

After poking around it some more I think it would be good to have the following:

1) individual HTTP/HTTPS proxy settings for every destination where juju is an HTTP client (image-stream, agent-stream, juju gui binaries, juju HA peer http connectivity, substrate HTTP API, charm store API);
2) individual HTTP/HTTPS settings for apt, snapd, lxd.

Right now we have a problem with using no-proxy even though it does not land in /etc/environment because we need to handle local connectivity (HA peers, client -> controller, substrate API) vs remote connectivity (cloud-images.ubuntu.com, streams.canonical.com) differently.

no-proxy just becomes too large for us to handle and we run out of stack memory.

apt proxy settings can be set separately but agent binaries have to be downloaded via a proxy (we do not always do mirrors).

agent-metadata-url only provides a way to specify a mirror, not a proxy. Mirrors (apt mirrors, snap store, image mirrors) and proxies are not mutually exclusive though so a tuple per http destination would be good.

no-proxy just becomes too large for us to use as it has to include hostnames and does not universally support CIDR.

I think we are not so worried about configuring deployed applications to use proxy settings via global options - this can be done by charms. As for our daemons, we could leverage CIDR and wildcard support for http(s) proxies in golang if it ever gets landed in addition to individual options:

juju - golang
snapd - golang
lxd - golang

https://go-review.googlesource.com/c/go/+/75730
https://go-review.googlesource.com/c/go/+/68091
https://github.com/golang/go/issues/16704

[Dmitrii Shcherbakov (dmitriis)]

As for the per-destination part, this can be done by using different Transports for different types of HTTP requests:

https://golang.org/pkg/net/http/#Transport
type Transport struct {
        // Proxy specifies a function to return a proxy for a given
        // Request.
...
        Proxy func(*Request) (*url.URL, error)

[Thanassis Zakopoulos (zakthan)]

Perfect for Centos7 OS. Thanks!

[Dmitrii Shcherbakov (dmitriis)]

I'm going to remove snapd from this bug as there is a separate one for the core snap proxy settings behavior:

https://bugs.launchpad.net/snapd/+bug/1791587 (the latest upstream patch mentioned in https://bugs.launchpad.net/snapd/+bug/1791587/comments/9 seems to work but the version of snapd with a fix has not been SRUed yet).