juju uses proxy to access bootstrap node

[Ashley Lai]

When using proxy in juju enviroment, juju uses this proxy to access bootstrap node and failed.

I added the following in ~/.juju/environments.yaml
    http-proxy: http://91.189.89.33:3128
    https-proxy: http://91.189.89.33:3128
    no-proxy: localhost,10.245.0.10

and got the following error:
  "4":
    agent-state: started
    agent-version: 1.24.2
    dns-name: hayward-35.oil
    instance-id: /MAAS/api/1.0/nodes/node-9fc3db80-c4cd-11e3-824b-00163efc5068/
    series: trusty
    containers:
      4/lxc/0:
        agent-state-info: 'failed to retrieve the template to clone: lxc container
          creation failed: error executing "lxc-create": + ''['' amd64 = i686 '']'';
          + ''['' 0 = 0 '']''; + case "$hostarch:$arch" in; + :; + ''['' tryreleased
          ''!='' daily -a tryreleased ''!='' released -a tryreleased ''!='' tryreleased
          '']''; + ''['' -z /var/lib/lxc/juju-trusty-lxc-template '']''; ++ id -u;
          + ''['' 0 ''!='' 0 '']''; + config=/var/lib/lxc/juju-trusty-lxc-template/config;
          + ''['' -z /var/lib/lxc/juju-trusty-lxc-template/rootfs '']''; + type ubuntu-cloudimg-query;
          ubuntu-cloudimg-query is /usr/bin/ubuntu-cloudimg-query; + type wget; wget
          is /tmp/wget084105148/wget; + cache=/var/cache/lxc/cloud-trusty; + ''[''
          0 -eq 1 '']''; + mkdir -p /var/cache/lxc/cloud-trusty; + ''['' tryreleased
          = tryreleased '']''; + stream=released; + ubuntu-cloudimg-query trusty released
          amd64; + ''['' -n https://10.245.0.177:17070/environment/3acdd43a-47b9-4604-8ea5-c5a65638d0e3/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + url2=https://10.245.0.177:17070/environment/3acdd43a-47b9-4604-8ea5-c5a65638d0e3/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          ++ basename https://10.245.0.177:17070/environment/3acdd43a-47b9-4604-8ea5-c5a65638d0e3/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + filename=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + ''['' -n https://10.245.0.177:17070/environment/3acdd43a-47b9-4604-8ea5-c5a65638d0e3/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + do_extract_rootfs; + cd /var/cache/lxc/cloud-trusty; + ''['' 0
          -eq 1 '']''; + trap wgetcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' ''!''
          -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz '']''; + wget https://10.245.0.177:17070/environment/3acdd43a-47b9-4604-8ea5-c5a65638d0e3/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          --2015-07-24 23:31:40-- https://10.245.0.177:17070/environment/3acdd43a-47b9-4604-8ea5-c5a65638d0e3/images/lxc/trusty/amd64/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          Connecting to 91.189.89.33:3128... connected.; Proxy tunneling failed: ForbiddenUnable
          to establish SSL connection.; + build_root_tgz ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + url=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + filename=; ++ mktemp
          -d -p .; + xdir=./tmp.RCZ3f1cYeT; ++ basename ubuntu-14.04-server-cloudimg-amd64-root.tar.gz;
          + tarname=ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + imgname=''trusty-*-cloudimg-amd64.img'';
          + trap buildcleanup EXIT SIGHUP SIGINT SIGTERM; + ''['' 0 -eq 1 -o ''!''
          -f /var/cache/lxc/cloud-trusty/ubuntu-14.04-server-cloudimg-amd64-root.tar.gz
          '']''; + rm -f ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + echo ''Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz''; Downloading
          cloud image from ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; + wget
          ubuntu-14.04-server-cloudimg-amd64-root.tar.gz; --2015-07-24 23:31:40-- http://ubuntu-14.04-server-cloudimg-amd64-root.tar.gz/;
          Connecting to 91.189.89.33:3128... connected.; Proxy request sent, awaiting
          response... 504 Gateway Time-out; 2015-07-24 23:31:40 ERROR 504: Gateway
          Time-out.; + echo ''Couldn''\''''t find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.'';
          Couldn''t find cloud image ubuntu-14.04-server-cloudimg-amd64-root.tar.gz.;
          + exit 1; + buildcleanup; + cd /var/lib/lxc/juju-trusty-lxc-template/rootfs;
          + umount -l /var/cache/lxc/cloud-trusty/./tmp.RCZ3f1cYeT; umount: /var/cache/lxc/cloud-trusty/./tmp.RCZ3f1cYeT:
          not mounted; + true; + rm -rf /var/cache/lxc/cloud-trusty; lxc_container:
          lxccontainer.c: create_run_template: 1125 container creation template for
          juju-trusty-lxc-template failed; lxc_container: lxc_create.c: main: 271
          Error creating container juju-trusty-lxc-template'
        instance-id: pending
        series: trusty
    hardware: arch=amd64 cpu-cores=8 mem=16384M tags=oil-slave-3,hardware-sm15k,hw-alai-staging,hw-glance-sm15k

10.245.0.10 - maas server
10.245.0.117 - bootstrap node

[Ashley Lai (alai)]

Adding the bootstrap node to no-proxy list may work but this should be done automatically.

[Cheryl Jennings (cherylj)]

I believe a workaround would be for you to add 10.245.0.177, and the IPs of any additional state servers you may have to the no-proxy list.

I'm still investigating the creation of the lxc containers to see if there's a way to specify to not use the proxy when creating them.

[Ian Booth (wallyworld)]

wget is used to grab the lxc image from the state server(s).
wget accepts a --no-proxy arg to disable use of any proxies configured via env vars or whatever.
So the wget script generated by the container code just needs to have this arg added.

[Cheryl Jennings (cherylj)]

Review request: http://reviews.vapour.ws/r/2267/

[Cheryl Jennings (cherylj)]

1.24 review request: http://reviews.vapour.ws/r/2293/

[John A Meinel (jameinel)]

Shouldn't we really be setting the local network (10.245.* in this case?). Consider any HTTP request between machines in the environment. The Admin should really not need to pay close attention to the IP addresses that the cloud assigns their machines and add them to the no-proxy list. That really seems like Juju's job.

[John A Meinel (jameinel)]

It looks like we'll go ahead with this fix and land wget --no-proxy into 1.24 and master. But we'll look to update our network model so that in whatever network space a given service/machine is in, it has a no-proxy setting the related subnets.

[Cheryl Jennings (cherylj)]

This problem does not apply to KVM instances, but if using a proxy when deploying with KVM, users must include "http://" or "https://" in their proxy settings, or else the deploy will fail with a vague exit code 1 error. This stems from the uvt-simplestreams-libvirt command returning this error when not using "http://" or "https://":

Proxy URLs must have explicit schemes

[Ashley Lai (alai)]

Bug to add all nodes' IPs no no-proxy list.

https://bugs.launchpad.net/juju-core/+bug/1488139

[Chris J Arges (arges)]

Hello Ashley, or anyone else affected,

Accepted juju-core into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/juju-core/1.22.8-0ubuntu1~14.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!