lxd containers not using configured proxy for downloading images

[Luca]

I installed MAAS 1.9 and juju 2.0-beta9, the Internet access is provided through proxy. Juju bootstrap correctly but is unable to create LXD container. The error was:

2016-06-17 10:24:17 ERROR juju.provisioner provisioner_task.go:677 cannot start instance for machine "0/lxd/0": failed to ensure LXD image: unable to get LXD image for ubuntu-xenial: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:443

I tried to copy metadata locally using juju metadata generate-image but it report the error:

ERROR cmd supercommand.go:448 model "controller" cannot provide region and endpoint

juju list-controllers
CONTROLLER MODEL USER CLOUD/REGION
devmaas-controller* default admin@local devmaas

juju list-models
MODEL OWNER STATUS LAST CONNECTION
controller admin@local available never connected
default* admin@local available 2 minutes ago

[Luca (l-dellefemmine)]

More detailed log from machine-0:

2016-06-21 08:57:02 INFO juju.provisioner container_initialisation.go:102 initial container setup with ids: [0/lxd/6]
2016-06-21 08:57:02 DEBUG lxd client.go:67 Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":[]}
2016-06-21 08:57:02 INFO juju.provisioner provisioner_task.go:402 found machine pending provisioning id:0/lxd/6, details:0/lxd/6
2016-06-21 08:57:02 INFO juju.provisioner provisioner_task.go:246 provisioner-harvest-mode is set to destroyed; unknown instances not stopped []
2016-06-21 09:01:19 DEBUG juju.provisioner.lxd broker.go:471 address allocation feature flag not enabled; using multi-bridge networking for container "0/lxd/6"
2016-06-21 09:01:29 DEBUG lxd client.go:67 Raw response: {"error":"not found","error_code":404,"type":"error"}
2016-06-21 09:01:29 DEBUG juju.tools.lxdclient client.go:172 connecting to LXD remote "cloud-images.ubuntu.com": "https://cloud-images.ubuntu.com/releases"
2016-06-21 09:01:30 INFO juju.tools.lxdclient client_image.go:135 found image from https://cloud-images.ubuntu.com/releases for xenial = 98dc4bd7cca2fcd24c82ee403baaca3a5bd1845f925540786d0954a5973c181c
2016-06-21 09:01:30 DEBUG lxd client.go:67 Posting {"auto_update":true,"public":false,"source":{"certificate":"","fingerprint":"98dc4bd7cca2fcd24c82ee403baaca3a5bd1845f925540786d0954a5973c181c","mode":"pull","protocol":"simplestreams","server":"https://cloud-images.ubuntu.com/releases","type":"image"}}
 to http://unix.socket/1.0/images
2016-06-21 09:01:30 DEBUG lxd client.go:67 Raw response: {"type":"async","status":"Operation created","status_code":100,"metadata":{"id":"7337a50b-f4c6-4f58-b369-be1fd1adb21c","class":"task","created_at":"2016-06-21T09:01:30.431790916Z","updated_at":"2016-06-21T09:01:30.431790916Z","status":"Running","status_code":103,"resources":null,"metadata":null,"may_cancel":false,"err":""},"operation":"/1.0/operations/7337a50b-f4c6-4f58-b369-be1fd1adb21c"}
2016-06-21 09:01:30 DEBUG lxd client.go:67 1.0/operations/7337a50b-f4c6-4f58-b369-be1fd1adb21c/wait
2016-06-21 09:01:50 DEBUG lxd client.go:67 Raw response: {"type":"sync","status":"Success","status_code":200,"metadata":{"id":"7337a50b-f4c6-4f58-b369-be1fd1adb21c","class":"task","created_at":"2016-06-21T09:01:30.431790916Z","updated_at":"2016-06-21T09:01:30.431790916Z","status":"Failure","status_code":400,"resources":null,"metadata":null,"may_cancel":false,"err":"Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:443"}}
2016-06-21 09:01:50 DEBUG juju.tools.lxdclient client_image.go:139 dropped 0 progress messages
2016-06-21 09:01:50 ERROR juju.provisioner provisioner_task.go:677 cannot start instance for machine "0/lxd/6": failed to ensure LXD image: unable to get LXD image for ubuntu-xenial: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:443

[James Tunnicliffe (dooferlad)]

I don't think our LXD code has anything to do with this and what is necessary is for the host to have the correct http and https proxies set. This should be done by our proxy updater code already.

Presumably if machine 0 needs a proxy this has already been set up in MAAS. Is that the case? Just having an APT proxy can get you a long way so it it isn't unusual to only notice problems after machine 0 seems to have bootstrapped correctly.

Can you check that on machine 0 /home/ubuntu/.juju-proxy exists and has the correct values? What does PRINTENV | grep PROXY show?

[Luca (l-dellefemmine)]

Hello James,
I configured MAAS proxy for APT, HTTP and HTTPS.

Juju bootstrap configuration is:
http-proxy: http://10.52.18.78:3128
https-proxy: http://10.52.18.78:3128
no-proxy: localhost,192.168.1.1,10.52.18.3
bootstrap-timeout: 3600
default-series: xenial

~$ cat /home/ubuntu/.juju-proxy:
export http_proxy=http://10.52.18.78:3128
export HTTP_PROXY=http://10.52.18.78:3128
export https_proxy=http://10.52.18.78:3128
export HTTPS_PROXY=http://10.52.18.78:3128
export no_proxy=10.52.18.3,127.0.0.1,192.168.1.1,192.168.1.101,::1,localhost
export NO_PROXY=10.52.18.3,127.0.0.1,192.168.1.1,192.168.1.101,::1,localhost

~$ printenv | grep PROXY
NO_PROXY=10.52.18.3,127.0.0.1,192.168.1.1,192.168.1.101,::1,localhost
HTTPS_PROXY=http://10.52.18.78:3128
HTTP_PROXY=http://10.52.18.78:3128

Juju can deploy application inside the physical machine, but fail to deploy the same application inside on lxd container.

The command juju add-machine lxd:0 fail too with the error described above

[John A Meinel (jameinel)]

LXD does have a configuration you can set without having to get its environment variables modified.

lxc config set core.proxy_http=
lxc config set core.proxy_https=
lxc config set core.proxy_ignore_hosts=

It will use the 'http_proxy' environment variable if core.proxy_http is not set.

In the short term you can use the above commands to set the http proxy. Juju itself should be setting those.

[Andreas Hasenack (ahasenack)]

John, this is a freshly provisioned machine, how would I add those proxy settings? Would I have to quickly ssh in as soon as it's reachable and race juju and set the proxy like that in lxd? Or did you have something else in mind?

[Tim Penhey (thumper)]

Once the machine is provisioned, what happens if you just ssh into the machine and try to start a lxd container manually?

[Luca (l-dellefemmine)]

I ssh into the machine and tried to start a new container:

~$ sudo -E lxc launch ubuntu:16.04
Creating sauceless-jona
error: Get https://cloud-images.ubuntu.com/releases/streams/v1/index.json: Unable to connect to: cloud-images.ubuntu.com:443

~$ printenv | grep PROXY
NO_PROXY=10.52.18.2,127.0.0.1,192.168.2.1,192.168.2.2,::1,localhost
FTP_PROXY=http://10.52.18.78:3128
HTTPS_PROXY=http://10.52.18.78:3128
HTTP_PROXY=http://10.52.18.78:3128

I also set the proxy into the lxd config like John said and tried again

~$ sudo -E lxc config set core.proxy_https 'http://10.52.18.78:3128'
~$ sudo -E lxc config set core.proxy_http 'http://10.52.18.78:3128'
~$ sudo -E lxc launch ubuntu:16.04
Creating soporiferous-lorene
Retrieving image: 100%
Starting soporiferous-lorene

~$ sudo -E lxc list
+---------------------+---------+------+------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------------------+---------+------+------+------------+-----------+
| soporiferous-lorene | RUNNING | | | PERSISTENT | 0 |
+---------------------+---------+------+------+------------+-----------+

[Tim Penhey (thumper)]

@Luca, if I can provide you with some juju binaries, can you test it to see if my fix works?

[Luca (l-dellefemmine)]

Hi Tim, of course I'll test it.

[Tim Penhey (thumper)]

Luca, you can find juju and jujud here:
   http://people.canonical.com/~tim/

You will need to bootstrap with --upload-tools.

Let me know how you go.

[Luca (l-dellefemmine)]

Hi Tim,
using your fix I can add a new container. The container is running, but juju report it in pending status.

~$ juju machines
MACHINE STATE DNS INS-ID SERIES AZ
0 started 10.15.1.2 4y3h7p xenial rack01
  0/lxd/2 pending juju-ef10a4-0-lxd-2 xenial

I ssh into the machine and check the services running in the container

~$ sudo -E lxc exec juju-ef10a4-0-lxd-2 ps awx
  PID TTY STAT TIME COMMAND
    1 ? Ss 0:00 /sbin/init
   45 ? Ss 0:00 /lib/systemd/systemd-udevd
   48 ? Ss 0:00 /lib/systemd/systemd-journald
   85 ? Ssl 0:00 /usr/lib/accountsservice/accounts-daemon
   86 ? Ss 0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
   90 ? Ssl 0:00 /usr/sbin/rsyslogd -n
   91 ? Ssl 0:00 /usr/lib/snapd/snapd
   92 ? Ss 0:00 /usr/sbin/cron -f
   94 ? Ss 0:00 /lib/systemd/systemd-logind
   95 ? Ss 0:00 /usr/sbin/atd -f
  102 ? Ssl 0:00 /usr/lib/policykit-1/polkitd --no-debug
  203 ? Ss 0:00 /usr/sbin/sshd -D
  225 console Ss+ 0:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 linux
  322 ? Rs+ 0:00 ps awx

There are not juju agent.

[Tim Penhey (thumper)]

Hmm... I'm pretty sure that is because using the proxy it can't talk to the controller. Now we need to find out why.

[Tim Penhey (thumper)]

Not propagating the no_proxy values, will update and push new fix.

[Tim Penhey (thumper)]

Binaries at http://people.canonical.com/~tim/ updated.

[Luca (l-dellefemmine)]

Hi Tim,
your new binaries propagate the no_proxy value correctly.

~$ sudo -E lxc config show
config:
  core.proxy_http: http://10.52.18.78:3128
  core.proxy_https: http://10.52.18.78:3128
  core.proxy_ignore_hosts: 10.52.18.2,127.0.0.1,192.168.2.1,192.168.2.2,::1,localhost

I added another container but the result was the same, the container is created and is running but juju agent isn't installed.

I tried again with other server without VLANs (the other one has 7 VLANs assigned) and I was able to create a container.

~$ juju machines
MACHINE STATE DNS INS-ID SERIES AZ
0 started 10.15.1.2 4y3h7p xenial rack01
  0/lxd/3 pending juju-ef10a4-0-lxd-3 xenial
2 started 192.168.2.106 4y3h7q xenial rack02
  2/lxd/0 started 192.168.2.3 juju-ef10a4-2-lxd-0 xenial

For my point of view the bug is fixed.

Thanks

[Felipe Reyes (freyes)]

Review request http://reviews.vapour.ws/r/5345/
github pull request https://github.com/juju/juju/pull/5907

[Alex (stojimir)]

I seem to be experiencing this same symptom ...

Trying to start a container on a machine with already deployed charm, fails to start a container with a different charm

Machine State DNS Inst id Series AZ Message
0 started 135.111.102.193 c637fs xenial default Deployed

neutron-gateway/0* blocked idle 0 135.111.102.193 Missing relations: messaging, network-service

0/lxd/1 down pending xenial can't get info for image 'juju/xenial/amd64': not found

Juju agent version 2.3.4
MAAS version: 2.3.1 (6470-g036d646-0ubuntu1~16.04.1

[Alex (stojimir)]

p.s. does not matter if charm is deployed or not, just trying to add container to a deployed machine fails

Machine State DNS Inst id Series AZ Message
0 started 135.111.102.193 c637fs xenial default Deployed
0/lxd/0 down pending xenial failed to start machine 0/lxd/0 (can't get info for image 'juju/xenial/amd64': not found), retrying in 10s (6 more attempts)

[Vidmantas (vidmantasvgtu)]

0/lxd/0 down pending bionic failed to start machine 0/lxd/0 (can't get info for image 'juju/bionic/amd64': not found), retrying in 10s (4 more attempts)
0/lxd/1 down pending bionic failed to start machine 0/lxd/1 (can't get info for image 'juju/bionic/amd64': not found), retrying in 10s (5 more attempts)
0/lxd/2 down pending bionic failed to start machine 0/lxd/2 (can't get info for image 'juju/bionic/amd64': not found), retrying in 10s (6 more attempts)
0/lxd/3 down pending bionic failed to start machine 0/lxd/3 (can't get info for image 'juju/bionic/amd64': not found), retrying in 10s (4 more attempts)
0/lxd/4 down pending bionic failed to start machine 0/lxd/4 (can't get info for image 'juju/bionic/amd64': not found), retrying in 10s (5 more attempts)

[Richard Harding (rharding)]

@Vidmantas what version of Juju are you using? Older versions of Juju have issues with bionic but the newer snaps should be ok here.