provider/lxd: cannot bootstrap when network is bridged

[Andrew Wilkins]

In Juju 2.0, the controllers connect to LXD using the address of the host with TLS auth. We determine the host address by obtaining the address of the default gateway. This only works when the gateway is the host, which is not the case when the network is bridged.

LXD may grow the ability to proxy unix sockets eventually, which would be preferable: the host would inject the LXD socket into the container, and the controller would use the unix socket method instead. This support doesn't exist yet.

In the mean time, we could have the bootstrap process identify the address of the host and inject that into the container as metadata. The container can query /dev/lxd to obtain that metadata. This is prone to the address becoming stale, but that is at unlikely, and can be fixed by the user by updating the container metadata manually.

[Spyderdyne (spyderdyne)]

This is currently unassigned and represents a complete show stopper for anyone attempting to evaluate Juju for enterprise adoption. Can someone point me to the variable name so I can replace the IP detection mechanism with a static value and run my own custom compiled version because this doesn't seem to be getting fixed any time soon.

I can assert that it is useful to actively developed projects like this one that providing a config file override option for internal variable substitution is generally frowned upon and complained about by developers, but somehow always gets used for something in production. That being said, if the variable for the node IP had such an override there would at least be a workaround.

As of right now there is no end in sight to this issue and nobody is assigned to work on it still. I personally had the option of adopting Juju orchestration for a client integration until I ran into this issue. You are losing the support of enterprise customers every day that this issue remains in existence.

Thanks.

[Dmitrii Shcherbakov (dmitriis)]

Hi Spyderdyne,

I can point you to this (the code is as of a83896d):

provider/lxd/environ_raw.go:
func newRawProvider(spec environs.CloudSpec) (*rawProvider, error) {
     client, err := newClient(spec, ioutil.ReadFile, utils.RunCommand)
     if err != nil {
          return nil, errors.Annotate(err, "creating LXD client")
}

Here is where it gets the default gateway:

newRawProvider -> newClient -> getRemoteConfig |

...

hostAddress, err := getDefaultGateway(runCommand)
if err != nil {
     return nil, errors.Annotate(err, "getting gateway address")
}

...

return &lxdclient.Config{
lxdclient.Remote{
     Name: "remote",
     Host: hostAddress,
...

[Dmitrii Shcherbakov (dmitriis)]

getDefaultGateway:

https://github.com/juju/juju/blob/a83896d913d7e43c960e441c1e41612116d92d46/provider/lxd/environ_raw.go#L155

[Andrew Wilkins (axwalk)]

Spyderdyne, please see https://github.com/juju/juju/commit/19bf802db6511d2081369da2a3fe9b13f1bcb9fd. With that change, you can define a "lxd" type cloud in clouds.yaml and explicitly specify the IP address of the LXD host as the endpoint.

[Andrew Wilkins (axwalk)]

(I'm leaving the bug open, as the patch above is a workaround. Ideally we would be poking the Unix socket into the container, but this requires changes to LXD itself.)

[Anastasia (anastasia-macmood)]

Please note that the above commit is in Juju 2.1.x no on 2.0.x

[Spyderdyne (spyderdyne)]

I will pull and test tomorrow when i get a chance. This is great because i
was not making any progress learning GOlang.

On Jan 11, 2017 8:55 PM, "Andrew Wilkins" <email address hidden>
wrote:

> Spyderdyne, please see
> https://github.com/juju/juju/commit/19bf802db6511d2081369da2a3fe9b
> 13f1bcb9fd.
> With that change, you can define a "lxd" type cloud in clouds.yaml and
> explicitly specify the IP address of the LXD host as the endpoint.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1640455
>
> Title:
> provider/lxd: cannot bootstrap when network is bridged
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1640455/+subscriptions
>

[Spyderdyne (spyderdyne)]

Sorry it took me so long to get back to this. I added the Dev repos and have a much later version now:

root@ayana-angel:~# juju --version
2.1-beta4-xenial-amd64

root@ayana-angel:~# !425
juju bootstrap localhost juju-lxd-0
Creating Juju controller "juju-lxd-0" on localhost/localhost
Looking for packaged Juju agent version 2.1-beta4 for amd64
To configure your system to better support LXD containers, please see: https://github.com/lxc/lxd/blob/master/doc/production-setup.md
Launching controller instance(s) on localhost/localhost...
 - juju-46012c-0 (arch=amd64)
Fetching Juju GUI 2.2.7
Waiting for address
Attempting to connect to 2601:cd:c100:4e00:216:3eff:fe9b:b89d:22
Attempting to connect to fc00::216:3eff:fe9b:b89d:22
Attempting to connect to 192.168.1.145:22
sudo: unable to resolve host juju-46012c-0
Logging to /var/log/cloud-init-output.log on the bootstrap machine
Running apt-get update
Running apt-get upgrade

Installing curl, cpu-checker, bridge-utils, cloud-utils, tmux
Fetching Juju agent version 2.1-beta4 for amd64
Installing Juju machine agent
Cloud-init v. 0.7.8 running 'init-local' at Mon, 16 Jan 2017 19:45:32 +0000. Up 15.0 seconds.
Cloud-init v. 0.7.8 running 'init' at Mon, 16 Jan 2017 19:48:06 +0000. Up 169.0 seconds.
ci-info: ++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++
ci-info: +--------+------+-----------------------------+---------------+-------+-------------------+
ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
ci-info: +--------+------+-----------------------------+---------------+-------+-------------------+
ci-info: | lo | True | 127.0.0.1 | 255.0.0.0 | . | . |
ci-info: | lo | True | ::1/128 | . | host | . |
ci-info: | eth0 | True | 192.168.1.145 | 255.255.255.0 | . | 00:16:3e:9b:b8:9d |
ci-info: | eth0 | True | fe80::216:3eff:fe9b:b89d/64 | . | link | 00:16:3e:9b:b8:9d |
ci-info: +--------+------+-----------------------------+---------------+-------+-------------------+
ci-info: +++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
ci-info: | 0 | 0.0.0.0 | 192.168.1.1 | 0.0.0.0 | eth0 | UG |
ci-info: | 1 | 192.168.1.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: +-------+-------------+-------------+---------------+-----------+-------+
Generating public/private rsa key pair.
Your identification has been saved in /etc/ssh/ssh_host_rsa_key.
Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub.
The key fingerprint is:
SHA256:B7+bNAtiEg+XE/VBCxc0TQUjrUqoCD9xTgsYkAfQeJc root@juju-46012c-0
The key's randomart image is:
+---[RSA 2048]----+
|*= . ooO=+o. |
|+ + E . + =o. |
| = . ... o. |
|o o o .o.o. |
| o Boo+.S.o |
| + += ... . |
| .. + ....

[Andrew Wilkins (axwalk)]

You will need to create a custom cloud definition for this workaround, in ~/.local/share/juju/clouds.yaml:

  clouds:
    lxd:
      type: lxd
      endpoint: <lxd host IP address>

The Juju controller will then contact LXD on that

[Spyderdyne (spyderdyne)]

Created with,

*root@ayana-angel:~/.local/share/juju# cat clouds.yaml *

*clouds:*

* ayana-angel:*

* type: lxd*

* endpoint: 192.168.1.220*

*root@ayana-angel:~/.local/share/juju# pwd*

*/root/.local/share/juju*

*root@ayana-angel:~/.local/share/juju# whoami*

*root*

*2017-01-17 02:40:20 ERROR cmd supercommand.go:458 new environ: creating
LXD client: Get https://192.168.1.1:8443/1.0
<https://192.168.1.1:8443/1.0>: Unable to connect to: 192.168.1.1:8443
<http://192.168.1.1:8443>*

*ERROR failed to bootstrap model: subprocess encountered error code 1*

*root@ayana-angel:~/.local/share/juju# juju --version*

*2.1-beta4-xenial-amd64*

No change in behavior with this beta version.

Thanks.

On Mon, Jan 16, 2017 at 7:36 PM, Andrew Wilkins <
<email address hidden>> wrote:

> You will need to create a custom cloud definition for this workaround,
> in ~/.local/share/juju/clouds.yaml:
>
> clouds:
> lxd:
> type: lxd
> endpoint: <lxd host IP address>
>
> The Juju controller will then contact LXD on that
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1640455
>
> Title:
> provider/lxd: cannot bootstrap when network is bridged
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1640455/+subscriptions
>

[Spyderdyne (spyderdyne)]

Updated per your example:

root@ayana-angel:~/.local/share/juju# cat clouds.yaml
clouds:
    lxd:
      type: lxd
      endpoint: 192.168.1.220

root@ayana-angel:~/.local/share/juju# juju bootstrap localhost juju-lxd-0
Creating Juju controller "juju-lxd-0" on localhost/localhost

Reverted to previous config:

root@ayana-angel:~/.local/share/juju# cat clouds.yaml
clouds:
    ayana-angel:
      type: lxd
      endpoint: 192.168.1.220

juju bootstrap (no options for interactive mode)

Clouds
aws
aws-china
aws-gov
ayana-angel
azure
azure-china
cloudsigma
google
joyent
localhost
rackspace

Select a cloud [localhost]: ayana-angel

Enter a name for the Controller [ayana-angel]: angel-juju-lxd-0

Creating Juju controller "angel-juju-lxd-0" on ayana-angel
Looking for packaged Juju agent version 2.1-beta4 for amd64
To configure your system to better support LXD containers, please see: https://github.com/lxc/lxd/blob/master/doc/production-setup.md
Launching controller instance(s) on ayana-angel...

Someday I will look at addressing that DNS error when we are not using localhost bind:

Fetching Juju GUI 2.2.7
Waiting for address
Attempting to connect to 2601:cd:c100:4e00:216:3eff:fe5d:db60:22
Attempting to connect to fc00::216:3eff:fe5d:db60:22
sudo: unable to resolve host juju-03fdc2-0

...
 - juju-03fdc2-0 (arch=amd64)
Fetching Juju GUI 2.2.7
Waiting for address
Attempting to connect to 2601:cd:c100:4e00:216:3eff:fe5d:db60:22
Attempting to connect to fc00::216:3eff:fe5d:db60:22
sudo: unable to resolve host juju-03fdc2-0
Logging to /var/log/cloud-init-output.log on the bootstrap machine
Running apt-get update
Running apt-get upgrade
Installing curl, cpu-checker, bridge-utils, cloud-utils, tmux
Fetching Juju agent version 2.1-beta4 for amd64
Installing Juju machine agent
Starting Juju machine agent (service jujud-machine-0)
Bootstrap agent now started
Contacting Juju controller at fc00::216:3eff:fe5d:db60 to verify accessibility...
Bootstrap complete, "angel-juju-lxd-0" controller now available.
Controller machines are in the "controller" model.
Initial model "default" added.

Success!!!

root@ayana-angel:~/.local/share/juju# lxc list
+---------------+---------+----------------------+---------------------------------------------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------------+---------+----------------------+---------------------------------------------+------------+-----------+
| chef0 | RUNNING | 192.168.1.103 (eth0) | 2601:cd:c100:4e00:216:3eff:fe5f:dca8 (eth0) | PERSISTENT | 0 |
| | | | fc00::216:3eff:fe5f:dca8 (eth0) | | |
+---------------+---------+----------------------+---------------------------------------------+------------+-----------+
| juju-03fdc2-0 | RUNNING | 192.168.1.136 (eth0) | 2601:cd:c100:4e00:216:3eff:fe5d:db60 (eth0) | PERSISTENT | 0 |
| | | | fc00::216:3eff:fe5d:db60 (eth0) | | |
+---------------+---------+--------------------...

[Andrew Wilkins (axwalk)]

Great, thanks for confirming :)
Obviously this isn't an ideal solution; we'll be working with the LXD folks at some stage to get the necessary changes in LXD and Juju so that all of this is automatic.

Patches very much welcome for fixing the name resolution error/warning.