deploying from charmstore fails on lxd provider in a restricted network

[Kevin W Monroe]

I am unable to deploy charms in a network restricted environment, even though I have http[s]_proxy and no_proxy set:

cat /etc/issue
Ubuntu Xenial Xerus (development branch) \n \l

juju version
2.0-beta3-xenial-ppc64el

export http_proxy=http://squid.internal:3128
export https_proxy=http://squid.internal:3128
export no_proxy=`echo localhost 10.245.67.130 10.0.3.{1..255} | sed 's/ /,/g'`

juju bootstrap admin-lxd lxd --upload-tools
...
2016-03-28 20:13:24 INFO cmd cmd.go:129 Bootstrap complete, local.admin-lxd now available.
2016-03-28 20:13:24 INFO cmd supercommand.go:454 command finished

juju deploy ubuntu
ERROR storing charm for URL "cs:trusty/ubuntu-6": cannot retrieve charm "cs:trusty/ubuntu-6": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/trusty/ubuntu-6/archive: dial tcp 162.213.33.122:443: getsockopt: connection timed out

I could wget the above 'archive' file from my host machine, but could not from the admin controller unit. I thought this would help:

juju set-model-config http-proxy=$http_proxy https-proxy=$https_proxy no-proxy=$no_proxy

After this, I verified the proxy settings were correct on unit 0, and I was able to "juju ssh 0" and wget the above 'archive' file successfully. Still, 'juju deploy ubuntu' failed in the same way. This time with debug:

juju deploy ubuntu --debug
2016-03-28 21:28:29 INFO juju.cmd supercommand.go:60 running juju [2.0-beta3 gc go1.6]
2016-03-28 21:28:29 INFO juju.api api.go:213 connecting to API addresses: [10.0.3.62:17070]
2016-03-28 21:28:29 INFO juju.api apiclient.go:492 dialing "wss://10.0.3.62:17070/model/713a88d3-b538-4cc0-8f2b-536881afd02d/api"
2016-03-28 21:28:29 INFO juju.api apiclient.go:269 connection established to "wss://10.0.3.62:17070/model/713a88d3-b538-4cc0-8f2b-536881afd02d/api"
2016-03-28 21:28:29 DEBUG juju.api api.go:361 API hostnames unchanged - not resolving
2016-03-28 21:28:29 DEBUG juju.api api.go:147 failed to connect via bootstrap config: aborted
2016-03-28 21:28:29 DEBUG httpbakery client.go:244 client do GET https://api.jujucharms.com/charmstore/v5/ubuntu/meta/any?include=id&include=supported-series {
2016-03-28 21:28:30 DEBUG httpbakery client.go:246 } -> error <nil>
2016-03-28 21:32:45 DEBUG juju.api apiclient.go:518 health ping failed: connection is shut down
2016-03-28 21:32:45 ERROR cmd supercommand.go:448 storing charm for URL "cs:trusty/ubuntu-6": cannot retrieve charm "cs:trusty/ubuntu-6": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/trusty/ubuntu-6/archive: dial tcp 162.213.33.121:443: getsockopt: connection timed out

[James Tunnicliffe (dooferlad)]

This is probably a duplicate of #1556207 and I have a fix landing for that at the moment in the 1.25 branch. It will then be forward ported to 2.0.

The specific problem of not being able to download a charm is because the juju clients try to download it via the API server, but the API server isn't on the no_proxy list, so it goes to the proxy, which can't see the API server...

You can find what Juju has set its proxy settings are by looking on any Juju deployed machine at ~/.juju-proxy. If you source that file (as generated by the current Juju binary) and wget works on a Juju machine that isn't machine 0 before I land my fix then this is a different problem.

[Kevin W Monroe (kwmonroe)]

Hey James, not sure if I'm parsing your comment correctly. I have my state server ip listed "no_proxy" on my juju client machine. Are you saying that the juju client ignores "no_proxy"? Here's where I'm getting confused:

"If you source that file ... and wget works on a Juju machine that isn't machine 0 ... then this is a different problem"

I can copy the trusty/ubuntu charm onto my net-restricted host (which has http[s]_proxy and no_proxy configured) and do this:

- juju deploy local:trusty/ubuntu
- juju run --unit ubuntu/0 'cat .juju_proxy' # <- looks good, just like my client machine
- juju run --unit ubuntu/0 'wget https://api.jujucharms.com/charmstore/v5/trusty/ubuntu-7/archive'
- wget https://api.jujucharms.com/charmstore/v5/trusty/ubuntu-7/archive

All those things work. Now I try:

- juju deploy trusty/ubuntu
ERROR storing charm for URL "cs:trusty/ubuntu-7": cannot retrieve charm "cs:trusty/ubuntu-7": cannot get archive: Get https://api.jujucharms.com/charmstore/v5/trusty/ubuntu-7/archive: dial tcp 162.213.33.122:443: getsockopt: connection timed out

Hopefully, I'm just not grok'ing your last para and this is indeed being fixed in bug 1556207.

[Cheryl Jennings (cherylj)]

Kevin - it could be the case that since you set the proxy information once the controller was bootstrapped, that those proxy changes didn't get applied to the currently running jujud processes. You can see this comment for some background: https://bugs.launchpad.net/juju-core/+bug/1556207/comments/15

We can see if that's the case by setting the proxy information in the config during bootstrap (either with --config http-proxy=<url>, etc., or by passing in a yaml file with the proxy info). If you do that, does it work for you?

[Kevin W Monroe (kwmonroe)]

Easy money Cheryl! This worked (after destroying my old controller):

$ juju bootstrap --upload-tools --debug --config http-proxy=$http_proxy --config https-proxy=$https_proxy --config no-proxy=$no_proxy admin-lxd lxd

$ juju get-model-config | grep proxy # <- looks good

$ juju deploy trusty/ubuntu
Added charm "cs:trusty/ubuntu-7" to the model.
Deploying charm "cs:trusty/ubuntu-7" with the charm series "trusty".

Thanks for the tip that setting proxy info post-bootstrap was problematic.