Rebooting the virtual machines breaks Juju networking

[Matt Bruzek]

I am using the KVM provider with 1.24.2-vivid-amd64.

I needed to reboot the virtual machines for a kernel update and was not able to connect to them after a reboot. The command 'juju status' reports their ip addresses have changed, but I can tell from virt-manager they still have the original ip addresses.

ID WORKLOAD-STATE AGENT-STATE VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE

consul/0 unknown idle 1.24.2.1 1 53/tcp,8302/tcp,8400/tcp,8500/tcp 192.168.122.5

docker-exp/0 error idle 1.24.2.1 2 172.17.42.1 hook failed: "network-relation-joined"
docker-exp/1 error idle 1.24.2.1 3 172.17.42.1 hook failed: "network-relation-joined"

On my machine 172.17.42.1 appears to be the docker0 device:
mbruzek@warhorse:~$ ifconfig
docker0 Link encap:Ethernet HWaddr b2:de:6e:10:dd:4e
          inet addr:172.17.42.1 Bcast:0.0.0.0 Mask:255.255.255.0
          inet6 addr: fe80::b0de:6eff:fe10:dd4e/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:127 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B) TX bytes:14830 (14.8 KB)

eth0 Link encap:Ethernet HWaddr 3c:97:0e:ea:19:ee
          inet addr:192.168.1.229 Bcast:192.168.1.255 Mask:255.255.255.0
          inet6 addr: fe80::3e97:eff:feea:19ee/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:1496314 errors:0 dropped:0 overruns:0 frame:0
          TX packets:862059 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1651547557 (1.6 GB) TX bytes:216351435 (216.3 MB)
          Interrupt:20 Memory:b4a00000-b4a20000
...

The KVM virtual machines should be within the default kvm network (192.168.122.2-192.168.122.254).

$ virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>06031329-a028-4bad-a0e8-7f716ac4f33f</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:a1:89:1f'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

I don't know what other logs or information you need. Please advise.

[Matt Bruzek (mbruzek)]

The state server machine log from the system where the error occurred.

[Matt Bruzek (mbruzek)]

After trial and error I could log in using 'ssh ubuntu@192.168.122.110' but only 'juju ssh docker-exp/0' was broken.

It looks like Juju lost track of the IP address. Again 172.17.42.1 is wrong here:

mbruzek@warhorse:~/workspace/charms/trusty/docker$ juju ssh docker-exp/1
Warning: Permanently added '172.17.42.1' (ECDSA) to the list of known hosts.
Permission denied (publickey,password).

[James Tunnicliffe (dooferlad)]

My fault - fixed this in trunk but not 1.24. Have cherry picked the change and just running the unit tests. Should land soon.