controller on a virtual machine, virbr0 from nova-compute being returned as primary ip address for host.

[David Britton]

This bug seems somehow similar to https://bugs.launchpad.net/juju/+bug/1567683

Juju ssh to base-machine/0 seems to work fine:

ubuntu@atlas:~$ juju ssh base-machine/0
Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-47-generic x86_64)

 * Documentation: https://help.ubuntu.com
 * Management: https://landscape.canonical.com
 * Support: https://ubuntu.com/advantage

  System information as of Thu Nov 24 05:50:28 UTC 2016

  System load: 1.39 Processes: 870
  Usage of /: 2.4% of 457.83GB Users logged in: 0
  Memory usage: 52% IP address for br-eth0: 10.96.11.42
  Swap usage: 0% IP address for lxdbr0: 10.0.0.1

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

Last login: Thu Nov 24 05:49:33 2016 from 10.96.0.10
ubuntu@bibarel:~$ logout
Connection to 10.96.11.42 closed.

But juju ssh to base-machine/2 is returning '192.168.122.1', which happens to be the same virtual network that libvirt chose on the machine that I'm doing the SSH *from*! leading to a ssh MITM error.

ubuntu@atlas:~$ juju ssh base-machine/2
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:eo18LNDZyXH7mfhuVO4Wof9+7rvTaYqvG182zhuMzic.
Please contact your system administrator.
Add correct host key in /tmp/ssh_known_hosts069250830 to get rid of this message.
Offending RSA key in /tmp/ssh_known_hosts069250830:7
  remove with:
  ssh-keygen -f "/tmp/ssh_known_hosts069250830" -R 192.168.122.1
ECDSA host key for 192.168.122.1 has changed and you have requested strict checking.
Host key verification failed.

----- Machine that base-machine/2 should have logged into -------

  System load: 0.0 Users logged in: 0
  Usage of /: 2.6% of 457.83GB IP address for br-eth0: 10.96.11.46
  Memory usage: 6% IP address for lxdbr0: 10.0.0.1
  Swap usage: 0% IP address for virbr0: 192.168.122.1
  Processes: 727

Notice the virbr0 above.

nova-compute-kvm, nova-compute-libvirt has been installed on this host (part of an openstack deployment). I believe the virbr0 is coming from there.

I think the tricky bit here, is the controller is deployed in a network that has 192.168.122.0/24 as routable (the controller is on a VM). And the virbr0 that is created when the kvm/libvirt packages are installed are also choosing that network.

This leads to a total failure. Agents get messed up, relations start trying to contact '192.168.122.1' to send relation data, but that internally routes to THEMSELVES. It's a very bad situation.

Truth be told, I'm not sure what should happen in this situation. But it took me a while to figure out what was going on here, so I thought I would file a bug.

[Adam Collard (adam-collard)]

Possible dupe of bug 1473069

[John A Meinel (jameinel)]

Fix should be in 2.1 and develop.
https://github.com/juju/juju/pull/6849
https://github.com/juju/juju/pull/6850