tools download in cloud-init should not go through http[s]_proxy

[David Britton]

GOOD: setting http_proxy in your environment worked for the bootstrap
GOOD: secondary machines that were launched also got the http proxy settings
BAD: they tried to download tools through the proxy from the bootstrap node

The curl in cloud-init should either run with no proxy, or the addresses of the bootstrap node should be added to the no_proxy setting (the later sounds flaky).

 + printf %s\n export http_proxy=http://squid.internal:3128
export HTTP_PROXY=http://squid.internal:3128
export https_proxy=http://squid.internal:3128
export HTTPS_PROXY=http://squid.internal:3128
export no_proxy=localhost,10.245.160.0,10.245.160.50,10.245.160.51,10.245.160.54,10.245.160.55,10.245.160.56,10.245.160.57,10.245.160.58,10.245.160.59,10.245.160.60,10.245.160.62,10.245.160.63,10.245.160.64,10.245.160.66,10.245.160.68,10.245.160.69,10.245.160.70,10.245.160.71,10.245.160.72,10.245.160.73,10.245.160.74,10.245.160.75,10.245.160.76,10.245.160.78,10.245.160.79,10.245.160.83,10.245.160.84,10.245.160.85,10.245.160.86,10.245.160.90,10.245.161.156,10.245.161.158,10.245.161.159,10.245.161.160,10.245.161.161,10.245.161.162
export NO_PROXY=localhost,10.245.160.0,10.245.160.50,10.245.160.51,10.245.160.54,10.245.160.55,10.245.160.56,10.245.160.57,10.245.160.58,10.245.160.59,10.245.160.60,10.245.160.62,10.245.160.63,10.245.160.64,10.245.160.66,10.245.160.68,10.245.160.69,10.245.160.70,10.245.160.71,10.245.160.72,10.245.160.73,10.245.160.74,10.245.160.75,10.245.160.76,10.245.160.78,10.245.160.79,10.245.160.83,10.245.160.84,10.245.160.85,10.245.160.86,10.245.160.90,10.245.161.156,10.245.161.158,10.245.161.159,10.245.161.160,10.245.161.161,10.245.161.162
+ chown ubuntu:ubuntu /home/ubuntu/.juju-proxy
+ mkdir -p /var/lib/juju/locks
+
+ id ubuntu
+ chown ubuntu:ubuntu /var/lib/juju/locks
+ mkdir -p /var/log/juju
+ chown syslog:adm /var/log/juju
+ bin=/var/lib/juju/tools/1.23-beta2-trusty-amd64
+ mkdir -p /var/lib/juju/tools/1.23-beta2-trusty-amd64
+ echo Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --insecure -o $bin/tools.tar.gz <[https://10.5.0.149:17070/tools/1.23-beta2-trusty-amd64]>
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --insecure -o $bin/tools.tar.gz <[https://10.5.0.149:17070/tools/1.23-beta2-trusty-amd64]>
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),102(netdev)
uid=1000(ubuntu) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev),102(netdev)
+ seq 5
+ printf Attempt 1 to download tools from %s...\n https://10.5.0.149:17070/tools/1.23-beta2-trusty-amd64
Attempt 1 to download tools from https://10.5.0.149:17070/tools/1.23-beta2-trusty-amd64...
+ curl -sSfw tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s --insecure -o /var/lib/juju/tools/1.23-beta2-trusty-amd64/tools.tar.gz https://10.5.0.149:17070/tools/1.23-beta2-trusty-amd64
curl: (22) The requested URL returned error: 403

The error is unclear a bit because of all the formatting params on the curl command (not sure why all that is there). This one is clearer:

ubuntu@juju-serverstack-machine-1:~$ curl -sS --insecure https://10.5.0.149:17070/tools/1.23-beta2-tru
sty-amd64
curl: (56) Received HTTP code 403 from proxy after CONNECT
ubuntu@juju-serverstack-machine-1:~$

[David Britton (dpb)]

Just so it's clear, the node 0 tools download needs to go through the proxy (of course). anything that tries to download from an internal address should not (like the other nodes).

[Eric Snow (ericsnowcurrently)]

So would something like "HTTP_PROXY= curl ..." work as the effective no-proxy one-off? If so, the fix in environs/cloudinit/cloudinit_ubuntu.go is a one-liner.

[Eric Snow (ericsnowcurrently)]

The catch is that the fix should not apply to bootstrap.

[Cheryl Jennings (cherylj)]

I talked with Eric about this and we had a couple thoughts:

1 - If the user goes through the trouble to specify an http proxy, I'm not sure it's appropriate for juju to "override" that setting, even for the bootstrap node. It should be up to the user to set up the proxy to handle private addresses. I believe this was Nate's thought on this too.

2 - If we decide that we want juju to override the proxy settings for the bootstrap node, it might be better to add it to the no_proxy list so that any future communication to the bootstrap node that happens doesn't also run into this problem.

I can start on a patch for point #2, but would like feedback on the first point as I think that's the more correct answer.

[David Britton (dpb)]

On Fri, Apr 03, 2015 at 03:24:33PM -0000, Cheryl Jennings wrote:
> I talked with Eric about this and we had a couple thoughts:
>
> 1 - If the user goes through the trouble to specify an http proxy, I'm
> not sure it's appropriate for juju to "override" that setting, even for
> the bootstrap node. It should be up to the user to set up the proxy to
> handle private addresses. I believe this was Nate's thought on this
> too.
>

In this particular case (getting tools from the bootstrap node), I'm
not sure what you have said makes sense.

1) An http proxy is generally (in every case I've seen) configured and
set up for talking to external networks. Not for proxying traffic on
the local subnet. That would seem like a weird use case to me -- not
something to optimize for. It would unecessarily create traffic on the
local net, and bottleneck on a proxy server for no real technical
benefit.

2) At least with openstack, and the way that Canonical is recommending
installing it, the addresses that is being tried here is the nodes
internal address. Those IPs are not routable outside of the neutron
network. Of course as a network admin you could do all kinds of things
to make those addresses routable, but by default they are not.

3) Theoretically speaking, even if juju were trying the bootstrap node's
externally reachable address, I would need to reconfigure the proxy
server -- which I don't have access to, it's a corporate controlled
thing, and getting them to add my local ip range to that server also
feels weird. If I were a network admin, I would probably not go for
that. I'm thinking they would tell me: "The proxy server exists to let
me reach the internet. Not to cache local network traffic."

4) I'm not a network admin, so this is just how I've used proxy servers
in the past. Might be good to ask someone in the know about this, like
elmo. I *do* at least know that the address you are trying for the
bootstrap node will not work to proxy as it's internal to the cloud
(point #2).

> 2 - If we decide that we want juju to override the proxy settings for
> the bootstrap node, it might be better to add it to the no_proxy list so
> that any future communication to the bootstrap node that happens doesn't
> also run into this problem.

I think adding internal ip addresses to no_proxy is a valid approach,
however I would caution about what happens if they change. Especially
with HA state, and those being able to move. I really don't know, but
it's what was in the back of my head when I called it "flaky" in the
original request.

--
David Britton <email address hidden>

[David Britton (dpb)]

OK, one last data point here, and I understand why this bugged me so much. This regressed from 1.20.x where I last tried this.

If you install 1.20.x from trusty, you will notice that the secondary nodes in the deployment all download from 'streams' (which can be proxied just fine). But now that tools download is routed through the bootstrap node, this failure is now evident to me.

[Cheryl Jennings (cherylj)]

Ah, thank you for that info. I was wondering if this was a regression because it surprised me that it hadn't been seen before.

And good point on the HA case. I think it might be better to do what Eric had suggested in comment #2 and, for the non-bootstrap node, not use a proxy when downloading the tools. I'll look into things a little more, and chat with him again.

[Cheryl Jennings (cherylj)]

Chatted with Nate and he had an excellent suggestion of adding a config option to let the user choose whether or not juju uses the proxy to connect to other machines in the environment, since there will most likely be people who want it both ways.

I will get started on implementing that, unless there are objections.

[Cheryl Jennings (cherylj)]

So, after even more discussions today, it was decided that we will just disable the proxy for the tools download when we're not the bootstrap node. There's a convenient --noproxy option for curl which I tested and it will actually disable the proxy for the curl command in question. I'll finish up the tests for this and push up the changes in the morning.

[mahmoh (mahmoh)]

Hi this at a _customer_ deployment, please increase the priority of this if possible, feel free to contact me if needed.

[Cheryl Jennings (cherylj)]

I have the code changes approved and am trying to get them merged. I'm running into problems with the CI tests that's leading to the delay.

[Katherine Cox-Buday (cox-katherine-e)]

This was also fixed in 1.22.3