snapd ignores proxy settings set via core snap

[Dmitrii Shcherbakov]

Setting proxy settings via the core snap does not have any effect.

sudo snap set core proxy.http=http://192.0.2.2:3128
sudo snap set core proxy.https=http://192.0.2.2:3128

Although I wouldn't want my global /etc/environment to be modified by snapd in any case but it is not modified:

cat /etc/environment
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
TESTVAR="testval"

When I go to my proxy server and check the logs there is nothing in them.

Looking at the implementation, the core snap proxy configuration handler only tries to update the global /etc/environment file - not the default HTTP transport used for snapd cases.

https://github.com/snapcore/snapd/blob/2.35.1/overlord/configstate/configcore/proxy.go#L72-L94

The default http transport is updated based on environment settings:

https://github.com/snapcore/snapd/blob/2.35.1/httputil/transport17.go#L33-L43
func newDefaultTransport() *http.Transport {
 // based on https://github.com/golang/go/blob/release-branch.go1.7/src/net/http/transport.go#L38
 return &http.Transport{
  Proxy: http.ProxyFromEnvironment,

https://golang.org/pkg/net/http/#ProxyFromEnvironment

Since those settings are not there, snapd doesn't use the proxy specified.

Instead, snapd needs to (thread-safely) set HTTP_PROXY, HTTPS_PROXY and NO_PROXY values in-memory based on core snap settings instead of trying to modify /etc/environment. For this to take effect connections to the snap store need to use new http.Transport objects every time they need to access it or react to in-memory environment variable changes.

apt policy snapd
snapd:
  Installed: 2.34.2+18.04
  Candidate: 2.34.2+18.04
  Version table:
 *** 2.34.2+18.04 500
        500 http://ru.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.32.5+18.04 500
        500 http://ru.archive.ubuntu.com/ubuntu bionic/main amd64 Packages

[Michael Vogt (mvo)]

Thanks for your bugreport.

Can you please try "sudo snap refresh --edge core" and see if that fixes the issue for you? We added support for looking up the http proxy from the core configuration on classic in master.

[Dmitrii Shcherbakov (dmitriis)]

Hi Michael,

It does work with the edge version on a classic system:

sudo snap refresh --edge core
2018-09-10T16:10:56+03:00 INFO Waiting for restart...
core (edge) 16-2.35.1+git944.b5355ba from Canonical✓ refreshed

snap find vlc
Name Version Publisher Notes Summary
vlc 3.0.4 videolan✓ - The ultimate media player
# ...

sudo tail -f /var/log/squid/access.log

1536585110.412 50341 10.10.10.218 TCP_TUNNEL/200 5122 CONNECT api.snapcraft.io:443 - HIER_DIRECT/91.189.92.40 -
1536585110.825 50383 10.10.10.218 TCP_TUNNEL/200 323505 CONNECT api.snapcraft.io:443 - HIER_DIRECT/91.189.92.40 -

It looks like those are the changes you mentioned:

https://github.com/snapcore/snapd/commit/56b7bd942d
https://github.com/snapcore/snapd/blob/56b7bd942d/httputil/client.go#L38-L47

[Dmitrii Shcherbakov (dmitriis)]

Maybe the bug status should be updated

old output:

 core (edge) 16-2.35.1+git944.b5355ba from Canonical✓ refreshed

It seems like the new core version for which everything worked well is now in stable:

➜ ~ snap info core | grep stable
  stable: 16-2.35.2 (5548) 92MB -

[Dmitrii Shcherbakov (dmitriis)]

Looks like there is still a problem even with the new core snap.

When the core snap is not installed snapd is still ignoring the proxy settings.

sudo snap install helm --classic
Ensure prerequisites for "helm" are available error: cannot perform the following tasks:
- Download snap "core" (5548) from channel "stable" (Get https://fastly.cdn.snapcraft.io/download-origin/fastly/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_5548.snap?token=1539302400_8cd4da523854cb54cecc8f8b322ab4560d9d2125: dial tcp 151.101.62.217:443: i/o timeout)

apt policy snapd
snapd:
  Installed: 2.34.2
  Candidate: 2.34.2
  Version table:
 *** 2.34.2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.0.2 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

[Calvin Hartwell (calvinh)]

This bug was previously logged and supposed to be fixed, but still exists: https://bugs.launchpad.net/juju/+bug/1737332

[Dmitrii Shcherbakov (dmitriis)]

Tried 2.35.4+18.04.

lxc launch ubuntu:bionic snaptest

# ... enable bionic proposed

root@snaptest:~# apt policy snapd
snapd:
  Installed: 2.35.4+18.04
  Candidate: 2.35.4+18.04
  Version table:
 *** 2.35.4+18.04 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

root@snaptest:~# ip route del default
root@snaptest:~# ip route
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.14
10.10.10.1 dev eth0 proto dhcp scope link src 10.10.10.14 metric 100

# the squid proxy itself works fine
root@snaptest:~# https_proxy=http://10.10.10.30:3128 curl https://api.snapcraft.io
snapcraft.io store API service - Copyright 2018 Canonical.

root@snaptest:~# snap set core proxy.http=http://10.10.10.30:3128 proxy.https=http://10.10.10.30:3128

root@snaptest:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.

root@snaptest:~# snap get core proxy.https
http://10.10.10.30:3128

root@snaptest:~# snap get core proxy.http
http://10.10.10.30:3128

strace -e connect -f -p `pgrep -f snapd` &

root@snaptest:~# snap install fcbtesting

[pid 1313] connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16 <unfinished ...>
[pid 1319] connect(5, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16 <unfinished ...>
[pid 1313] <... connect resumed> ) = 0
[pid 1319] <... connect resumed> ) = 0
[pid 1319] connect(4, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("91.189.92.41")}, 16) = -1 ENETUNREACH (Network is unreachable)
[pid 1319] connect(4, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("91.189.92.40")}, 16) = -1 ENETUNREACH (Network is unreachable)
[pid 1319] connect(4, {sa_family=AF_INET, sin_port=htons(9),

After a discussion on IRC it became apparent that the functionality works on Ubuntu core systems because they have a core snap pre-installed and not on classic systems.

[Dmitrii Shcherbakov (dmitriis)]

Built 2.36 from source, installed it in a LXD container and performed another test (http proxy is on a host to which there is a route in the routing table while there is no default gateway). Still getting the same behavior.

root@snaptest:~# snap --version
snap 2.36
snapd 2.36
series 16
ubuntu 18.04
kernel 4.15.0-34-generic

# removed the default route
root@snaptest:~# ip route
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.14
10.10.10.1 dev eth0 proto dhcp scope link src 10.10.10.14 metric 100
10.208.111.0/24 dev lxdbr0 proto kernel scope link src 10.208.111.1 linkdown

root@snaptest:~# snap get core proxy.https
http://10.10.10.30:3128

root@snaptest:~# snap get core proxy.http
http://10.10.10.30:3128

udo snap install --edge core --classic
[pid 24750] connect(4, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
[pid 24754] connect(9, {sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("127.0.0.53")}, 16) = 0
[pid 24744] connect(4, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("91.189.92.40")}, 16) = -1 ENETUNREACH (Network is unreachable)
[pid 24744] connect(4, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("91.189.92.20")}, 16) = -1 ENETUNREACH (Network is unreachable)
[pid 24744] connect(4, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("91.189.92.19")}, 16) = -1 ENETUNREACH (Network is unreachable)
[pid 24744] connect(4, {sa_family=AF_INET, sin_port=htons(9), sin_addr=inet_addr("91.189.92.41")}, 16) = -1 ENETUNREACH (Network is unreachable)

[Dmitrii Shcherbakov (dmitriis)]

Based on some experimentation I confirmed the following:

0) default gateway is disabled, proxy is accessible but is not used by snapd;
1) enable default gateway;
2) install the core snap;
3) disable default gateway;
4) proxy settings are used by snapd.

[Michael Vogt (mvo)]

Thanks @dmitriis - while we had a test for this it was not good enough, i.e. we did not test that this also works with an empty state. The test is added in https://github.com/snapcore/snapd/pull/6062 and the test highlights that there are indeed problems with some store code that uses http requests without going through the proxy. This is fixed in 6062 and we plan to cherry-pick this in 2.36.1 which will be released soon.

[Dmitrii Shcherbakov (dmitriis)]

# Seems to work with snapd built from source, 7bb38874f3bc8bbcac951cd51055579fc2c54473

root@snaptest:~# ip route
default via 10.10.10.1 dev eth0 proto dhcp src 10.10.10.14 metric 100
10.10.10.0/24 dev eth0 proto kernel scope link src 10.10.10.14
10.10.10.1 dev eth0 proto dhcp scope link src 10.10.10.14 metric 100
10.208.111.0/24 dev lxdbr0 proto kernel scope link src 10.208.111.1 linkdown

root@snaptest:~# ip route del default

root@snaptest:~# dpkg -i parts/snapd/snapd_2.36_amd64.deb
Selecting previously unselected package snapd.
(Reading database ... 45059 files and directories currently installed.)
Preparing to unpack .../snapd/snapd_2.36_amd64.deb ...
Unpacking snapd (2.36) ...
Setting up snapd (2.36) ...
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.autoimport.service → /lib/systemd/system/snapd.autoimport.service.
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.core-fixup.service → /lib/systemd/system/snapd.core-fixup.service.
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
Created symlink /etc/systemd/system/cloud-final.service.wants/snapd.seeded.service → /lib/systemd/system/snapd.seeded.service.
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.service → /lib/systemd/system/snapd.service.
Created symlink /etc/systemd/system/timers.target.wants/snapd.snap-repair.timer → /lib/systemd/system/snapd.snap-repair.timer.
Created symlink /etc/systemd/system/sockets.target.wants/snapd.socket → /lib/systemd/system/snapd.socket.
Created symlink /etc/systemd/system/final.target.wants/snapd.system-shutdown.service → /lib/systemd/system/snapd.system-shutdown.service.
snapd.failure.service is a disabled or a static unit, not starting it.
snapd.snap-repair.service is a disabled or a static unit, not starting it.

Processing triggers for mime-support (3.60ubuntu1) ...
Processing triggers for man-db (2.8.3-2) ...

root@snaptest:~# snap version
snap 2.36+git439.g7bb3887
snapd 2.36+git439.g7bb3887
series 16
ubuntu 18.04
kernel 4.15.0-34-generic

root@snaptest:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.

root@snaptest:~# snap set core proxy.http=http://10.10.10.30:3128 proxy.https=http://10.10.10.30:3128

root@snaptest:~# snap list
No snaps are installed yet. Try 'snap install hello-world'.

root@snaptest:~# snap install core
2018-11-09T06:27:39Z INFO Waiting for restart...
core 16-2.35.5 from Canonical✓ installed

/var/log/squid/access.log:

1541744717.062 378473 10.10.10.14 TCP_TUNNEL/200 141418518 CONNECT fastly.cdn.snapcraft.io:443 - HIER_DIRECT/151.101.2.217 -
1541744859.824 25487 10.10.10.14 TCP_TUNNEL/200 92285957 CONNECT fastly.cdn.snapcraft.io:443 - HIER_DIRECT/151.101.2.217 -
1541744859.824 31860 10.10.10.14 TCP_TUNNEL/200 16213 CONNECT api.snapcraft.io:443 - HIER_DIRECT/91.189.92.20 -
1541744859.824 31867 10.10.10.14 TCP_TUNNEL/200 5131 CONNECT api.snapcraft.io:443 - HIER_DIRECT/91.189.92.20 -
1541744859.824 25488 10.10.10.14 TCP_TUNNEL/200 6782 CONNECT api.snapcraft.io:443 - HIER_DIRECT/91.189.92.20 -
1541744859.824 30147 10.10.10.14 TCP_TUNNEL/200 348489 CONNECT api...

[Michael Vogt (mvo)]

Thanks for the report and for testing master. This should now work in 2.36.1 which is currently in the candiate channel and will be SRUed shortly.

[Dmitrii Shcherbakov (dmitriis)]

Added snapd package to track the SRU work.