juju-core

2.0 beta6: only able to access LXD containers (on maas deployed host) from the maas network

Bug #1576674 reported by Larry Michel
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Fix Released
Critical
Dimiter Naydenov
Canonical Juju 2.0-beta9
juju-core
Won't Fix
High
Unassigned
1.25
Won't Fix
High
Unassigned

Bug Description

I am deploying a bundle using maas as provider and I can only get to my LXD containers from the maas network. When on an external network, I have to go first to a server on the maas server with the right public key to be able to ssh into the LXD container.

This breaks our automated log collection for LXD containers.

This is from juju show-status:
===============================================================
[Units]
ID WORKLOAD-STATUS JUJU-STATUS VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
ceilometer/0 waiting idle 2.0-beta6 4/lxd/0 8777/tcp 10.244.192.237 Incomplete relations: identity
ceph/0 active idle 2.0-beta6 0 10.244.192.199 Unit is ready and clustered
cinder/0 waiting idle 2.0-beta6 1 10.244.192.212 Incomplete relations: identity, database
glance/0 waiting idle 2.0-beta6 1/lxd/0 9292/tcp 10.244.192.223 Incomplete relations: identity, database
heat/0 waiting idle 2.0-beta6 1/lxd/1 8000/tcp,8004/tcp 10.244.192.229 Incomplete relations: identity, database
keystone/0 waiting idle 2.0-beta6 2/lxd/0 10.244.192.219 Incomplete relations: database
mongodb/0 unknown idle 2.0-beta6 3/lxd/0 27017/tcp,27019/tcp,27021/tcp,28017/tcp 10.244.192.231
mysql/0 error idle 2.0-beta6 0/lxd/0 3306/tcp 10.244.192.217 hook failed: "shared-db-relation-joined" for nova-cloud-controller:shared-db
neutron-api/0 waiting idle 2.0-beta6 4/lxd/1 9696/tcp 10.244.192.221 Incomplete relations: identity, database
neutron-gateway/0 active idle 2.0-beta6 2 10.244.192.201 Unit is ready
nova-cloud-controller/0 waiting idle 2.0-beta6 3 3333/tcp,8773/tcp,8774/tcp 10.244.192.210 Incomplete relations: identity, database
nova-compute/0 error idle 2.0-beta6 4 10.244.192.214 hook failed: "install"
openstack-dashboard/0 waiting idle 2.0-beta6 4/lxd/2 80/tcp,443/tcp 10.244.192.227 Incomplete relations: identity
rabbitmq-server/0 active idle 2.0-beta6 1/lxd/2 5672/tcp 10.244.192.233 Unit is ready
swift-proxy/0 blocked idle 2.0-beta6 2/lxd/1 8080/tcp 10.244.192.225 Not enough storage zones for minimum replicas
swift-storage/0 error idle 2.0-beta6 4/lxd/3 10.244.192.235 hook failed: "swift-storage-relation-joined" for swift-proxy:swift-storage
===============================================================

If I try to ping all these addresses from my system:
===============================================================
jenkins@s9-lmic-trusty:~$ for i in 10.244.192.237 10.244.192.199 10.244.192.212 10.244.192.223 10.244.192.229 10.244.192.219 10.244.192.231 10.244.192.217 10.244.192.221 10.244.192.201 10.244.192.210 10.244.192.214 10.244.192.227 10.244.192.233 10.244.192.225 10.244.192.235; do ping -c 1 $i; done
PING 10.244.192.237 (10.244.192.237) 56(84) bytes of data.

--- 10.244.192.237 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.199 (10.244.192.199) 56(84) bytes of data.
64 bytes from 10.244.192.199: icmp_seq=1 ttl=60 time=1.93 ms

--- 10.244.192.199 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.937/1.937/1.937/0.000 ms
PING 10.244.192.212 (10.244.192.212) 56(84) bytes of data.
64 bytes from 10.244.192.212: icmp_seq=1 ttl=60 time=1.03 ms

--- 10.244.192.212 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.030/1.030/1.030/0.000 ms
PING 10.244.192.223 (10.244.192.223) 56(84) bytes of data.

--- 10.244.192.223 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.229 (10.244.192.229) 56(84) bytes of data.

--- 10.244.192.229 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.219 (10.244.192.219) 56(84) bytes of data.

--- 10.244.192.219 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.231 (10.244.192.231) 56(84) bytes of data.

--- 10.244.192.231 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.217 (10.244.192.217) 56(84) bytes of data.

--- 10.244.192.217 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.221 (10.244.192.221) 56(84) bytes of data.

--- 10.244.192.221 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.201 (10.244.192.201) 56(84) bytes of data.
64 bytes from 10.244.192.201: icmp_seq=1 ttl=60 time=1.35 ms

--- 10.244.192.201 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.351/1.351/1.351/0.000 ms
PING 10.244.192.210 (10.244.192.210) 56(84) bytes of data.
64 bytes from 10.244.192.210: icmp_seq=1 ttl=60 time=1.04 ms

--- 10.244.192.210 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.046/1.046/1.046/0.000 ms
PING 10.244.192.214 (10.244.192.214) 56(84) bytes of data.
64 bytes from 10.244.192.214: icmp_seq=1 ttl=60 time=1.11 ms

--- 10.244.192.214 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.110/1.110/1.110/0.000 ms
PING 10.244.192.227 (10.244.192.227) 56(84) bytes of data.

--- 10.244.192.227 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.233 (10.244.192.233) 56(84) bytes of data.

--- 10.244.192.233 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.225 (10.244.192.225) 56(84) bytes of data.

--- 10.244.192.225 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

PING 10.244.192.235 (10.244.192.235) 56(84) bytes of data.

--- 10.244.192.235 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
===============================================================

If I do the ping on system on the maas network:
===============================================================
jenkins@s9-lmic-trusty:~$ juju ssh 0
Warning: Permanently added '10.244.192.199' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)

 * Documentation: https://help.ubuntu.com/

  System information as of Fri Apr 29 12:44:18 UTC 2016

  System load: 0.0 Users logged in: 0
  Usage of /: 9.8% of 49.08GB IP address for br-eth0: 10.244.192.199
  Memory usage: 24% IP address for br-eth1: 10.244.226.74
  Swap usage: 0% IP address for lxcbr0: 10.0.3.1
  Processes: 129 IP address for lxdbr0: 10.0.4.1

  Graph this data and manage this system at:
    https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

Last login: Fri Apr 29 12:44:20 2016 from 10.245.162.103
ubuntu@puritanvm03:~$ for i in 10.244.192.237 10.244.192.199 10.244.192.212 10.244.192.223 10.244.192.229 10.244.192.219 10.244.192.231 10.244.192.217 10.244.192.221 10.244.192.201 10.244.192.210 10.244.192.214 10.244.192.227 10.244.192.233 10.244.192.225 10.244.192.235; do ping -c 1 $i; done
PING 10.244.192.237 (10.244.192.237) 56(84) bytes of data.
64 bytes from 10.244.192.237: icmp_seq=1 ttl=64 time=0.669 ms

--- 10.244.192.237 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.669/0.669/0.669/0.000 ms
PING 10.244.192.199 (10.244.192.199) 56(84) bytes of data.
64 bytes from 10.244.192.199: icmp_seq=1 ttl=64 time=0.035 ms

--- 10.244.192.199 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.035/0.035/0.035/0.000 ms
PING 10.244.192.212 (10.244.192.212) 56(84) bytes of data.
64 bytes from 10.244.192.212: icmp_seq=1 ttl=64 time=0.582 ms

--- 10.244.192.212 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.582/0.582/0.582/0.000 ms
PING 10.244.192.223 (10.244.192.223) 56(84) bytes of data.
64 bytes from 10.244.192.223: icmp_seq=1 ttl=64 time=0.617 ms

--- 10.244.192.223 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.617/0.617/0.617/0.000 ms
PING 10.244.192.229 (10.244.192.229) 56(84) bytes of data.
64 bytes from 10.244.192.229: icmp_seq=1 ttl=64 time=0.459 ms

--- 10.244.192.229 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.459/0.459/0.459/0.000 ms
PING 10.244.192.219 (10.244.192.219) 56(84) bytes of data.
64 bytes from 10.244.192.219: icmp_seq=1 ttl=64 time=0.556 ms

--- 10.244.192.219 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.556/0.556/0.556/0.000 ms
PING 10.244.192.231 (10.244.192.231) 56(84) bytes of data.
64 bytes from 10.244.192.231: icmp_seq=1 ttl=64 time=0.511 ms

--- 10.244.192.231 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.511/0.511/0.511/0.000 ms
PING 10.244.192.217 (10.244.192.217) 56(84) bytes of data.
64 bytes from 10.244.192.217: icmp_seq=1 ttl=64 time=0.527 ms

--- 10.244.192.217 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.527/0.527/0.527/0.000 ms
PING 10.244.192.221 (10.244.192.221) 56(84) bytes of data.
64 bytes from 10.244.192.221: icmp_seq=1 ttl=64 time=0.561 ms

--- 10.244.192.221 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.561/0.561/0.561/0.000 ms
PING 10.244.192.201 (10.244.192.201) 56(84) bytes of data.
64 bytes from 10.244.192.201: icmp_seq=1 ttl=64 time=0.369 ms

--- 10.244.192.201 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.369/0.369/0.369/0.000 ms
PING 10.244.192.210 (10.244.192.210) 56(84) bytes of data.
64 bytes from 10.244.192.210: icmp_seq=1 ttl=64 time=0.651 ms

--- 10.244.192.210 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.651/0.651/0.651/0.000 ms
PING 10.244.192.214 (10.244.192.214) 56(84) bytes of data.
64 bytes from 10.244.192.214: icmp_seq=1 ttl=64 time=0.365 ms

--- 10.244.192.214 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.365/0.365/0.365/0.000 ms
PING 10.244.192.227 (10.244.192.227) 56(84) bytes of data.
64 bytes from 10.244.192.227: icmp_seq=1 ttl=64 time=0.504 ms

--- 10.244.192.227 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.504/0.504/0.504/0.000 ms
PING 10.244.192.233 (10.244.192.233) 56(84) bytes of data.
64 bytes from 10.244.192.233: icmp_seq=1 ttl=64 time=1.04 ms

--- 10.244.192.233 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.048/1.048/1.048/0.000 ms
PING 10.244.192.225 (10.244.192.225) 56(84) bytes of data.
64 bytes from 10.244.192.225: icmp_seq=1 ttl=64 time=0.420 ms

--- 10.244.192.225 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.420/0.420/0.420/0.000 ms
PING 10.244.192.235 (10.244.192.235) 56(84) bytes of data.
64 bytes from 10.244.192.235: icmp_seq=1 ttl=64 time=0.573 ms

--- 10.244.192.235 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.573/0.573/0.573/0.000 ms
ubuntu@puritanvm03:~$
===============================================================

I am attaching some net config file and route info for one container. I am also including route info for host in the attachment.

These are the pkg versions for lxd/lxc:

ubuntu@puritanvm07:~$ dpkg -l|grep lxc
ii liblxc1 1.1.5-0ubuntu3~ubuntu14.04.1 amd64 Linux Containers userspace tools (library)
ii lxc 1.1.5-0ubuntu3~ubuntu14.04.1 amd64 Linux Containers userspace tools
ii lxc-templates 1.1.5-0ubuntu3~ubuntu14.04.1 amd64 Linux Containers userspace tools (templates)
ii lxcfs 2.0.0-0ubuntu2~ubuntu14.04.1 amd64 FUSE based filesystem for LXC
ii python3-lxc 1.1.5-0ubuntu3~ubuntu14.04.1 amd64 Linux Containers userspace tools (Python 3.x bindings)
ubuntu@puritanvm07:~$ dpkg -l|grep lxd
ii lxd 2.0.0-0ubuntu1~ubuntu14.04.1 amd64 Container hypervisor based on LXC - daemon
ii lxd-client 2.0.0-0ubuntu1~ubuntu14.04.1 amd64 Container hypervisor based on LXC - client

and maas version:

ubuntu@maas-integration-september:~$ dpkg -l |grep maas
ii maas 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS server all-in-one metapackage
ii maas-cli 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS command line API tool
ii maas-cluster-controller 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS server cluster controller
ii maas-common 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS server common files
ii maas-dhcp 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS DHCP server
ii maas-dns 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS DNS server
ii maas-proxy 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS Caching Proxy
ii maas-region-controller 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS server complete region controller
ii maas-region-controller-min 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS Server minimum region controller
ii python-django-maas 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS server Django web framework
ii python-maas-client 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS python API client
ii python-maas-provisioningserver 1.9.1+bzr4543-0ubuntu1~trusty1 all MAAS server provisioning libraries

and juju:

jenkins@s9-lmic-trusty:~/nexenta/builds/3$ dpkg -l|grep -m 2 juju
ii juju 2.0-beta6-0ubuntu1~14.04.1~juju1 all next generation service orchestration system
ii juju-2.0 2.0-beta6-0ubuntu1~14.04.1~juju1 amd64 Juju is devops distilled - client

See original description
Revision history for this message
Larry Michel (lmic) wrote :
description: updated
Revision history for this message
Larry Michel (lmic) wrote :
Download full text (7.4 KiB)

There seems to be additional networking issue which could be related to this. I am including here unless it can be determined that it is a totally separate issue.

I did an apt-get update and it was hanging for a while at:

0% [Connecting to archive.ubuntu.com (91.189.92.201)] [Connecting to security.u

before it eventually completed.

But yesterday, it had timed out during deployment:

2016-04-28 16:43:44 INFO install Reading package lists...
2016-04-28 16:43:45 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Installing ['debconf-utils', 'python-mysqldb', 'uuid', 'pwgen', 'dnsutils'] with options: ['--option=Dpkg::Options::=--force-confold']"]
2016-04-28 16:43:45 INFO juju-log Installing ['debconf-utils', 'python-mysqldb', 'uuid', 'pwgen', 'dnsutils'] with options: ['--option=Dpkg::Options::=--force-confold']
2016-04-28 16:43:45 INFO install Reading package lists...
2016-04-28 16:43:45 INFO install Building dependency tree...
2016-04-28 16:43:45 INFO install Reading state information...
2016-04-28 16:43:45 INFO install dnsutils is already the newest version.
2016-04-28 16:43:45 INFO install The following packages were automatically installed and are no longer required:
2016-04-28 16:43:45 INFO install libfreetype6 os-prober
2016-04-28 16:43:45 INFO install Use 'apt-get autoremove' to remove them.
2016-04-28 16:43:45 INFO install The following extra packages will be installed:
2016-04-28 16:43:45 INFO install libmysqlclient18 libossp-uuid16 mysql-common
2016-04-28 16:43:45 INFO install Suggested packages:
2016-04-28 16:43:45 INFO install python-egenix-mxdatetime mysql-server-5.1 mysql-server python-mysqldb-dbg
2016-04-28 16:43:45 INFO install The following NEW packages will be installed:
2016-04-28 16:43:45 INFO install debconf-utils libmysqlclient18 libossp-uuid16 mysql-common pwgen
2016-04-28 16:43:45 INFO install python-mysqldb uuid
2016-04-28 16:43:52 INFO juju.worker.leadership tracker.go:182 mysql/0 will renew mysql leadership at 2016-04-28 16:44:22.964080373 +0000 UTC
2016-04-28 16:44:22 INFO juju.worker.leadership tracker.go:182 mysql/0 will renew mysql leadership at 2016-04-28 16:44:52.964286114 +0000 UTC
2016-04-28 16:44:52 INFO juju.worker.leadership tracker.go:182 mysql/0 will renew mysql leadership at 2016-04-28 16:45:22.964495714 +0000 UTC
2016-04-28 16:45:22 INFO juju.worker.leadership tracker.go:182 mysql/0 will renew mysql leadership at 2016-04-28 16:45:52.964732572 +0000 UTC
2016-04-28 16:45:46 INFO install 0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
2016-04-28 16:45:46 INFO install Need to get 780 kB of archives.
2016-04-28 16:45:46 INFO install After this operation, 4266 kB of additional disk space will be used.
2016-04-28 16:45:46 INFO install Err http://archive.ubuntu.com/ubuntu/ trusty-updates/main mysql-common all 5.5.49-0ubuntu0.14.04.1
2016-04-28 16:45:46 INFO install Could not connect to archive.ubuntu.com:80 (91.189.92.201), connection timed out
2016-04-28 16:45:46 INFO install Err http://archive.ubuntu.com/ubuntu/ trusty/main debconf-utils all 1.5.51ubuntu2
2016-04-28 16:45:46 INFO install Unable to connect to archive.ubuntu.com:http:
2016-04-28 16:45:46 INFO...

Read more...

Revision history for this message
Larry Michel (lmic) wrote :

Previous comment applies to another LXD container in the same deployment: mysql/0 error idle 2.0-beta6 0/lxd/0 3306/tcp 10.244.192.217

Revision history for this message
Martin Packman (gz) wrote :

I think this is somewhat expected if you are using straight ssh and trying to get to 10. addresses on a bridge local to a particular maas machine.

Does using `juju ssh --proxy 1/lxd/0` for instance not work for log collection on containers?

Changed in juju-core:
status: New → Incomplete
tags: added: lxd maas-provider ssh
Revision history for this message
Larry Michel (lmic) wrote :

Yes, it does work after specifying the --proxy (juju ssh 1/lxd/0 doesn't do anything):

jenkins@s9-lmic-trusty:~/nexenta/builds/3$ juju ssh --proxy 1/lxd/0
Warning: Permanently added '10.244.192.187' (ECDSA) to the list of known hosts.
Warning: Permanently added '10.244.192.219' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)

 * Documentation: https://help.ubuntu.com/

 System information disabled due to load higher than 2.0

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@juju-machine-1-lxd-0:~$

Changed in juju-core:
status: Incomplete → New
Revision history for this message
Cheryl Jennings (cherylj) wrote :

I would expect that since the default behavior for containers is to get a sticky IP from MAAS, that you would be able to connect to any container from the same hosts that can connect to the MAAS nodes.

I wasn't able to recreate the problem locally on either beta6 or tip of master. In both cases, I could juju ssh directly to the container without using --proxy.

However, that was on xenial, and I see you're on trusty, and that may be a key difference. Will try with that tomorrow.

Revision history for this message
James Tunnicliffe (dooferlad) wrote :

There was a bug introduced recently where routes could be wiped out during network configuration. The fix for this has landed in 1.25 and I am porting it to 2.0 today. This is what is causing the apt issue (can't get to the internet because there isn't a default route). If previously you had routes set up to get to the 10.244.192.0/24 machines from your machine that was probably broken by that bug.

Revision history for this message
James Tunnicliffe (dooferlad) wrote :

Ignore that - the bug wasn't introduced into 2.0. Did this work with an earlier beta?

The routes don't look obviously wrong. I will try to recreate it on my system.

Revision history for this message
James Tunnicliffe (dooferlad) wrote :

I can't tell from the information that you have given, but I suspect that you have two return paths for packets from the containers.

I don't have the same network topology as you - I only have:
[my machine] <--> switch <--> [MAAS machines]

It sounds like you have:
[your machine] <--> gateway <--> switch <--> [MAAS machines]

So, unsurprisingly, I don't have any problems getting to my guests since my machine is on the same subnet as the MAAS ones. The routing tables for guests only have one route configured per IP address they are responding to and one default route:

ubuntu@juju-machine-0-lxc-0:~$ ip -d route
unicast default via 192.168.1.1 dev eth0 proto boot scope global
unicast 192.168.1.0/24 dev eth1 proto boot scope link
unicast 192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.109

This container has two NICs:
ubuntu@juju-machine-0-lxc-0:~$ ip -d addr
<snip loopback>
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:b0:0c:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
    veth
    inet 192.168.1.108/24 brd 192.168.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:feb0:ca2/64 scope link
       valid_lft forever preferred_lft forever
18: eth1@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:52:78:3d brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0
    veth
    inet 192.168.1.109/24 brd 192.168.1.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe52:783d/64 scope link
       valid_lft forever preferred_lft forever

The second route (without the src) will return traffic received on eth1, while the third route will return traffic received on eth0.

It looks like you have an extra route in 4/lxd/3 so you can return traffic on eth0 or eth1 - unfortunately "route -n" isn't much use here because it doesn't tell us about source interfaces.

If your gateway is dropping traffic that is responding on a different MAC address that the connection was initiated on, that would explain the problem you are seeing.

I don't know if Juju has done anything to the routing tables that would cause this. /etc/network/interfaces, if parsed one interface at a time, should be fine. I don't remember why we go to so much trouble with pre and post rules to set up addresses and routes, but it looks odd.

Changed in juju-core:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Andrew McDermott (frobware) wrote :

WIP - https://github.com/frobware/juju/tree/master-lp1576674

Mostly the same patch I gave to Larry earlier today but needs re-verification.

Cheryl Jennings (cherylj)
Changed in juju-core:
milestone: none → 2.0-beta7
Revision history for this message
Andrew McDermott (frobware) wrote :

I need to take another look at this patch in light of:

    https://bugs.launchpad.net/juju-core/+bug/1581074

Curtis Hovey (sinzui)
Changed in juju-core:
milestone: 2.0-beta7 → 2.0-beta8
Cheryl Jennings (cherylj)
Changed in juju-core:
milestone: 2.0-beta8 → 2.0-beta9
Andrew McDermott (frobware)
Changed in juju-core:
assignee: nobody → Dimiter Naydenov (dimitern)
Dimiter Naydenov (dimitern)
Changed in juju-core:
status: Triaged → In Progress
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

Fix proposed: https://github.com/juju/juju/pull/5512

Revision history for this message
Dimiter Naydenov (dimitern) wrote :

The proposed fix addresses this and several related issues, confirmed to affect multiple stakeholders.

tags: added: blocker
Changed in juju-core:
importance: High → Critical
Dimiter Naydenov (dimitern)
Changed in juju-core:
status: In Progress → Fix Committed
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

Related bug 1590689 - the fix needs backporting to 1.25.

Cheryl Jennings (cherylj)
tags: removed: blocker
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
Revision history for this message
Dimiter Naydenov (dimitern) wrote :

The duplicated addresses on juju-br0 and eth0 are due to curtin < 389 and cloud-init ignoring the /etc/cloud/cloud.cfg setting to disable generating fallback network config. Related bug: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1563296

In 1.25 it was confirmed the proposed fix is not necessary.

Canonical Juju QA Bot (juju-qa-bot)
affects: juju-core → juju
Changed in juju:
milestone: 2.0-beta9 → none
milestone: none → 2.0-beta9
Canonical Juju QA Bot (juju-qa-bot)
Changed in juju-core:
importance: Undecided → High
status: New → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.