bootstrap node does not use the proxy to fetch tools from streams.c.c

[Andreas Hasenack]

I have juju 1.25.0 bootstrapped on a cloud that sits behind a proxy. It was bootstrapped like this:
  lsci-cloud:
    http-proxy: http://10.245.200.25:8000
    https-proxy: http://10.245.200.25:8000
    no-proxy: localhost,127.0.0.1,10.245.200.22,10.245.200.25,10.245.200.1
    # source your cloud credentials before using this one
    default-series: trusty
    type: openstack
    admin-secret: secret
    auth-mode: userpass
    use-floating-ip: true
    network: user_net

bootstrap worked. Then another unit was added, and that is stuck:
 juju status --format=tabular
[Services]
NAME STATUS EXPOSED CHARM
block-storage-broker unknown false cs:~davidpbritton/trusty/block-storage-broker-0
postgresql unknown false cs:precise/postgresql-96
storage false cs:precise/storage-6

[Units]
ID WORKLOAD-STATE AGENT-STATE VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
block-storage-broker/0 unknown idle 1.25.0 0 10.245.205.143
postgresql/0 unknown allocating 1 10.245.205.144 Waiting for agent initialization to finish

[Machines]
ID STATE VERSION DNS INS-ID SERIES HARDWARE
0 started 1.25.0 10.245.205.143 8bf1964b-54fa-4f9a-83c3-a90efa871bee trusty arch=amd64 cpu-cores=1 mem=2048M root-disk=20480M availability-zone=region1-1
1 pending 10.245.205.144 87d5e6d6-4b3a-4a09-9d45-38ec358c9332 precise arch=amd64 cpu-cores=1 mem=2048M root-disk=20480M availability-zone=region1-1

Further inspection shows that the unit gave up trying to download the tools from the bootstrap node (10.100.0.4):
+ printf Attempt 5 to download tools from %s...\n https://10.100.0.4:17070/tools/1.25.0-precise-amd64
Attempt 5 to download tools from https://10.100.0.4:17070/tools/1.25.0-precise-amd64...
+ curl -sSfw tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --noproxy * --insecure -o /var/lib/juju/tools/1.25.0-precise-amd64/tools.tar.gz https://10.100.0.4:17070/tools/1.25.0-precise-amd64
curl: (22) The requested URL returned error: 400
tools from https://10.100.0.4:17070/tools/1.25.0-precise-amd64 downloaded: HTTP 400; time 130.742s; size 0 bytes; speed 0.000 bytes/s + [ 5 -lt 5 ]

Trying that curl command indeed fails.

I then inspected the bootstrap node, and found this in all-machines.log:
machine-0: 2015-11-11 14:19:00 ERROR juju.apiserver tools.go:58 GET(/tools/1.25.0-precise-amd64?%3Aversion=1.25.0-precise-amd64&) failed: error fetching tools: Get https://streams.canonical.com/juju/tools/releases/juju-1.25.0-precise-amd64.tgz: dial tcp 91.189.88.141:443: connection timed out

That shows it's trying to reach the streams host directly, without the proxy. That won't work.

juju get-env confirms the proxy settings:
$ juju get-env|grep -i proxy
http-proxy: http://10.245.200.25:8000
https-proxy: http://10.245.200.25:8000
no-proxy: localhost,127.0.0.1,10.245.200.22,10.245.200.25,10.245.200.1
proxy-ssh: true

So I don't know why the bootstrap node didn't honor the https-proxy setting. Other hits at https://streams.canonical.com seemed to have worked:
machine-0: 2015-11-11 14:15:05 DEBUG juju.environs.simplestreams simplestreams.go:429 read metadata index at "https://streams.canonical.com/juju/tools/streams/v1/index2.sjson"
...
machine-0: 2015-11-11 14:15:07 DEBUG juju.environs.simplestreams simplestreams.go:968 metadata: &{map[com.ubuntu.juju:12.04:amd64:{ 1.15.0 amd64 map[20151029:0xc210697660]} com.ubuntu.juju:14.10:i38 ......

I see hits for streams.canonical.com in the proxy logs to confirm that:
1447252393.167 585 10.245.205.143 TCP_MISS/200 25238 CONNECT streams.canonical.com:443 - FIRSTUP_PARENT/91.189.89.33 -

So maybe just the code that fetches tools forgot to include proxy support.

[Andreas Hasenack (ahasenack)]

all-machines.log from the bootstrap node

[Andreas Hasenack (ahasenack)]

cloud-init-output.log from the second instance which is stalled.

[Andreas Hasenack (ahasenack)]

Normal juju status output

[Andreas Hasenack (ahasenack)]

nova list, that shows the internal ips

[Nate Finch (natefinch)]

Fix landing here: https://github.com/juju/utils/pull/186

We were, indeed, ignoring the proxies set in the environment. I'm kind of surprised no one noticed this before now.

[Nate Finch (natefinch)]

The above PR has landed, but another one is needed for this to be truly in and fixed for juju. That PR is here: https://github.com/juju/juju/pull/4108

Unfortunately, master is currently blocked, but once it is unblocked, this will land.

[Cheryl Jennings (cherylj)]

Looks like backporting this fix would require us to branch juju/utils as a lot has changed since the current version in dependencies.tsv and the commit from https://github.com/juju/utils/pull/186

[Cheryl Jennings (cherylj)]

Forked utils for 1.25. PR for utils change: https://github.com/juju/utils/pull/195