[2.0 rc2] IPV6 address used for public address for Xenial machines with vsphere as provider

[Larry Michel]

Per conversation with Michael, this is different from bug 1624495. Looking at Xenial and Trusty VM, I agree.

Juju registers fe80::1 for all xenial nodes including the controller even though all these VMs have an IPV4 address and a list of IPV6 addresses as reported by vCenter. However, juju is reporting the IP from the lxdbr0 bridge which only has IPV6 addresses.

MODEL CONTROLLER CLOUD/REGION VERSION
default lmic-vsphere-rc2 vsphere/dc0 2.0-rc2
...

MACHINE STATE DNS INS-ID SERIES AZ
0 started fe80::1 juju-da2fbd-0 xenial
1 started 10.245.62.83 juju-da2fbd-1 trusty
2 started 10.245.62.84 juju-da2fbd-2 trusty
3 started fe80::1 juju-da2fbd-3 xenial
4 started fe80::1 juju-da2fbd-4 xenial
5 started fe80::1 juju-da2fbd-5 xenial
6 started 10.245.62.90 juju-da2fbd-6 trusty
7 started fe80::1 juju-da2fbd-7 xenial
8 started fe80::1 juju-da2fbd-8 xenial
9 started fe80::1 juju-da2fbd-9 xenial
10 started fe80::1 juju-da2fbd-10 xenial
11 started fe80::1 juju-da2fbd-11 xenial

This is the network config for juju-da2fbd-10:

ubuntu@ubuntuguest:~$ ifconfig
docker0 Link encap:Ethernet HWaddr 02:42:77:5e:a9:f7
          inet addr:10.1.45.1 Bcast:0.0.0.0 Mask:255.255.255.0
          UP BROADCAST MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

ens192 Link encap:Ethernet HWaddr 00:50:56:87:6d:ea
          inet addr:10.245.62.89 Bcast:10.245.63.255 Mask:255.255.192.0
          inet6 addr: fe80::250:56ff:fe87:6dea/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:114658 errors:0 dropped:0 overruns:0 frame:0
          TX packets:32950 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:287973902 (287.9 MB) TX bytes:19106951 (19.1 MB)

flannel.1 Link encap:Ethernet HWaddr 86:54:85:5a:76:73
          inet addr:10.1.45.0 Bcast:0.0.0.0 Mask:255.255.0.0
          inet6 addr: fe80::8454:85ff:fe5a:7673/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:8 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:192 errors:0 dropped:0 overruns:0 frame:0
          TX packets:192 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:13504 (13.5 KB) TX bytes:13504 (13.5 KB)

lxdbr0 Link encap:Ethernet HWaddr 62:91:83:76:f7:ae
          inet6 addr: fe80::1/64 Scope:Link
          inet6 addr: fe80::6091:83ff:fe76:f7ae/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B) TX bytes:470 (470.0 B)

By comparison, this is for trusty VM, there's no lxdbr0:

ubuntu@ubuntuguest:~$ ifconfig
eth0 Link encap:Ethernet HWaddr 00:50:56:87:8c:a1
          inet addr:10.245.62.90 Bcast:10.245.63.255 Mask:255.255.192.0
          inet6 addr: fe80::250:56ff:fe87:8ca1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:53512 errors:0 dropped:0 overruns:0 frame:0
          TX packets:41441 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:93668813 (93.6 MB) TX bytes:7508066 (7.5 MB)

lo Link encap:Local Loopback
          inet addr:127.0.0.1 Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING MTU:65536 Metric:1
          RX packets:50576 errors:0 dropped:0 overruns:0 frame:0
          TX packets:50576 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:10053617 (10.0 MB) TX bytes:10053617 (10.0 MB)

Attaching log from controller and Xenial machine.

[Michael Foord (mfoord)]

Do you have logs available?

[Larry Michel (lmic)]

Yes, they're attached to comment #1: logs.tar.gz. It has the /var/log from the controller and /var/log + /etc from the xenial vm.

[Larry Michel (lmic)]

Alexis, this is a blocking bug for Xenial. A bundle will deploy but the services will be unusable because DNS is broken. All the config files show fe80::1.

For example, this is with kubernetes:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURPekNDQWlPZ0F3SUJBZ0lKQU9LMEdNd0FpLzgxTUEwR0NTcUdTSWIzRFFFQkN3VUFNQmd4RmpBVUJnTlYKQkFNTURURXdMakkwTlM0Mk1pNHhNekl3SGhjTk1UWXhNREEwTVRVek9UTXhXaGNOTWpZeE1EQXlNVFV6T1RNeApXakFZTVJZd0ZBWURWUVFEREEweE1DNHlORFV1TmpJdU1UTXlNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DCkFROEFNSUlCQ2dLQ0FRRUF0TjhvNi9CNWRTdEhNNjUwYjJSVmt5WFN5bmtHT0JUajkwRnY3blh0R1luUDlZT2kKOHBGV2ZlYnZwSXJSa1cycUc0Z1h1bi92c2VXQ1FGelROUzNjYXVEQ2U1QmZpTFluOXBhNGJhM1lnb1diQjlRZwppcjRoNUlXQmlhZ0VIR1o4ZUxBMUVMK3pwQlhlSlBKNnE2OFFEZGxoWmFERzJpMW1oNEh6dS9qLzdDVFBMZzVMCm5zeEIvRHNreEkwbXhDUGpXZHNhbVEwMDNMSGgyVlRiWXF5c2I3dHVJMWtmS1NDa0NGMFcySDBrR2ExbyszaE4KY1d5Ylh2WmlycjgzSElsVy9lWlh6N3htNHlSSnRDaExwS3FGQzZma3BnRUMveVlKSFVSc2M1ZXhFaGlMRjdpdgphM3FFOW9hS3NzNW1mZ2Y1QVRYMm9rekYzOTJkWlVoeU5BSlI3d0lEQVFBQm80R0hNSUdFTUIwR0ExVWREZ1FXCkJCUkNCVlMzSjAvaFBrUUhNaEZseEI2L1NIT28rVEJJQmdOVkhTTUVRVEEvZ0JSQ0JWUzNKMC9oUGtRSE1oRmwKeEI2L1NIT28rYUVjcEJvd0dERVdNQlFHQTFVRUF3d05NVEF1TWpRMUxqWXlMakV6TW9JSkFPSzBHTXdBaS84MQpNQXdHQTFVZEV3UUZNQU1CQWY4d0N3WURWUjBQQkFRREFnRUdNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUNsCkMzR3NyRHlhTXlGYUZ5ZHNVRzJiWW8wOU5udXJZZyt2U0lWKzVkOFdKZ3ZVWVQ5Tk1mcEszdyt5enpNZXRWZFMKNzZ3S0FpZU14VE9LNGxoWkNQOVBDcTVnUDRSNkNyRzQrenIwdVRoUUIySWlhNXltNTBFL1dQNmF5WEJuMEg2ZwpBa0NPU2FXeW45SkhTRmJUd0FtbFJFNFBDU0RuRnNZeEpyc2FrYXRuWnArSXMrTFJTb3BIYjh0REZhSVM4cUZhClI5YzZYZDZZRkZLcC9XaHplNys5UXBkOXZrelVacVVwRUdlQUVKcFFYc002eWQzR0dhU0I2Y28yem8vaHFjK0oKU2E0aUlCQm1QTndtTkJ4V3pQczdFSXdOWlNlU0p4d2lNRm5yS1hRa0EzSXpQZlFjS2IxUm5BaXIydjFzRTZqbApZZlNXUHdDMDU1R2xKQW5SL0ExcQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0t
    server: https://fe80::1:443
  name: juju-cluster
contexts:
- context:
    cluster: juju-cluster
    user: ubuntu
  name: juju-context
current-context: juju-context
kind: Config
preferences: {}
users:
- name: ubuntu
  user:

[Michael Foord (mfoord)]

I can finally repro this. What I'm seeing is that machine addresses are never set, which is why a machine local address is being returned (it's all that juju knows about). I'm digging in to work out why, but my guess is that this is a vsphere specific problem.

[Larry Michel (lmic)]

One issue that I see which is not present in trusty is that lxdbr0 is configured by default even when there is no lxd container placement. The address seems to be coming from this lxd bridge.

[Michael Foord (mfoord)]

I've found the bug and it's easy to fix. The vsphere provider is explicitly marking this link-local address as public - so juju picks it as the best public address. Switching to "derived scope" solves the problem and I can deploy units, see that they get ipv4 addresses, and ssh into them all. PR to follow shortly.

[Michael Foord (mfoord)]

https://github.com/juju/juju/pull/6475