[2.4.x][2.5.x] bootstrap logic doesn't handle model-config proxy settings properly

[Dmitrii Shcherbakov]

Although juju-http-proxy and juju-https-proxy are passed to the bootstrap command as model-configs (via --config), Juju does not pass them down to the curl command used in cloud-init userdata used before agents are created. This results in the following output on provisioned controller machines in /var/log/cloud-init-output.log:

Fetching Juju agent version 2.4.7 for amd64
Attempt 1 to download agent binaries from https://streams.canonical.com/juju/tools/agent/2.4.7/juju-2.4.7-ubuntu-amd64.tgz...
curl: (7) Failed to connect to streams.canonical.com port 443: No route to host
agent binaries from https://streams.canonical.com/juju/tools/agent/2.4.7/juju-2.4.7-ubuntu-amd64.tgz downloaded: HTTP 000; time 16.527768s; size 0 bytes; spee
d 0.000 bytes/s Download failed, retrying in 15s
Attempt 2 to download agent binaries from https://streams.canonical.com/juju/tools/agent/2.4.7/juju-2.4.7-ubuntu-amd64.tgz...
curl: (7) Failed to connect to streams.canonical.com port 443: No route to host
agent binaries from https://streams.canonical.com/juju/tools/agent/2.4.7/juju-2.4.7-ubuntu-amd64.tgz downloaded: HTTP 000; time 0.004448s; size 0 bytes; speed
 0.000 bytes/s Download failed, retrying in 15s
Attempt 3 to download agent binaries from https://streams.canonical.com/juju/tools/agent/2.4.7/juju-2.4.7-ubuntu-amd64.tgz...
curl: (7) Failed to connect to streams.canonical.com port 443: No route to host

Relevant code:
https://github.com/juju/juju/blob/juju-2.4.7/cloudconfig/userdatacfg_unix.go#L436-L443
https://github.com/juju/juju/blob/juju-2.4.7/cloudconfig/userdatacfg_unix.go#L463-L466
https://github.com/juju/juju/blob/juju-2.4.7/cloudconfig/userdatacfg_unix.go#L38
https://github.com/juju/juju/blob/juju-2.4.7/cloudconfig/userdatacfg_unix.go#L46-L57
    printf "Attempt $n to download agent binaries from %s...\n" {{shquote .}}

Reproducer for localhost provider (considering there is a proxy on a directly connected subnet):

cat modelconfig-proxy-nomirror.yaml
cloudinit-userdata: |
  preruncmd:
    - ip route del default
  postruncmd:
    - ip route del default
juju-http-proxy: http://192.0.2.30:3128
juju-https-proxy: http://192.0.2.30:3128
apt-http-proxy: http://192.0.2.30:8000
apt-https-proxy: http://192.0.2.30:8000

juju bootstrap localhost --config=modelconfig-proxy-nomirror.yaml --no-gui --debug
14:19:32 INFO juju.cmd supercommand.go:56 running juju [2.4.7 gc go1.10.4]
# ...

Attempting to connect to 192.0.2.196:22
14:20:39 DEBUG juju.provider.common bootstrap.go:575 connection attempt for 192.0.2.196 failed: /var/lib/juju/nonce.txt does not exist
14:20:44 INFO cmd bootstrap.go:344 Connected to 192.0.2.196
14:20:44 INFO juju.cloudconfig userdatacfg_unix.go:465 Fetching agent: curl -sSfw 'agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.4.7/juju-2.4.7-ubuntu-amd64.tgz]>
14:20:44 INFO cmd bootstrap.go:414 Running machine configuration script...

[Dmitrii Shcherbakov (dmitriis)]

Suggested fix (first pass):

https://github.com/juju/juju/pull/9550

15:47:39 INFO cmd bootstrap.go:344 Connected to 192.0.2.231
15:47:39 INFO juju.cloudconfig userdatacfg_unix.go:484 Fetching agent: curl -sSfw 'agent binaries from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 --proxy http://192.0.2.30:3128 --noproxy 127.0.0.1,localhost,::1 -o $bin/tools.tar.gz <[https://streams.canonical.com/juju/tools/agent/2.4.7/juju-2.4.7-ubuntu-amd64.tgz]>
15:47:39 INFO cmd bootstrap.go:414 Running machine configuration script...

Although it gets me forward, even though --no-gui was used Juju fails at a different step.

2018-12-07 12:50:51 DEBUG juju.storage managedstorage.go:294 managed resource entry created with path "buckets/63fffe4d-9b22-4181-86c7-e5fceeccf50e/tools/2.4.7-bionic-amd64-1812f497766d67c64a93dfa7f4dab7e16701cdea4b83f583fbc88ba2ea5493f5" -> "c6411aa8ac4f651a01cf46608b1d3a77f312ff7b5b1780b39c63c6b1c82ebdbdcd2e5fa2b40dcf4e4ebdcaf695ba6bf8"
2018-12-07 12:50:51 WARNING juju.cmd.jujud bootstrap.go:321 cannot set up Juju GUI: cannot fetch GUI info: GUI metadata not found
2018-12-07 12:50:51 DEBUG juju.state open.go:215 closed state without error
2018-12-07 12:50:51 INFO cmd supercommand.go:465 command finished
2018-12-07 12:50:51 DEBUG juju.cmd.jujud main.go:201 jujud complete, code 0, err <nil>
Starting Juju machine agent (service jujud-machine-0)
Created symlink /etc/systemd/system/jujud-machine-0.service → /lib/systemd/system/jujud-machine-0/jujud-machine-0.service.
Created symlink /etc/systemd/system/multi-user.target.wants/jujud-machine-0.service → /lib/systemd/system/jujud-machine-0/jujud-machine-0.service.
RTNETLINK answers: No such process
15:50:52 ERROR juju.cmd.juju.commands bootstrap.go:540 failed to bootstrap model: subprocess encountered error code 2
15:50:52 DEBUG juju.cmd.juju.commands bootstrap.go:541 (error details: [{github.com/juju/juju/cmd/juju/commands/bootstrap.go:638: failed to bootstrap model} {subprocess encountered error code 2}])
15:50:52 DEBUG juju.cmd.juju.commands bootstrap.go:1144 cleaning up after failed bootstrap
15:50:52 INFO juju.provider.common destroy.go:21 destroying model "controller"
15:50:52 INFO juju.provider.common destroy.go:32 destroying instances
15:50:54 INFO juju.provider.common destroy.go:56 destroying storage
15:50:54 DEBUG juju.provider.lxd environ.go:193 instances: []
15:50:54 INFO cmd supercommand.go:465 command finished

[Dmitrii Shcherbakov (dmitriis)]

Addressed the GUI part in an updated PR (GUI is fetched using a separate curl invocation).

The following error actually happens for a different reason: I had postruncmd with `ip route del default` in postruncmd as well which was causing this problem.

RTNETLINK answers: No such process
15:50:52 ERROR juju.cmd.juju.commands bootstrap.go:540 failed to bootstrap model: subprocess encountered error code 2

Without it the error goes away.

[Dmitrii Shcherbakov (dmitriis)]

I'm adding ~field-high here because this is a major bug (we use proxies all the time in the field). The fix is provided here: https://github.com/juju/juju/pull/9550 (tested on a localhost provider with the procedure above to kill networking and only use squid for proxied internet access).