2.0 beta6: only able to access LXD containers (on maas deployed host) from the maas network
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Canonical Juju |
Fix Released
|
Critical
|
Dimiter Naydenov | ||
juju-core |
Won't Fix
|
High
|
Unassigned | ||
1.25 |
Won't Fix
|
High
|
Unassigned |
Bug Description
I am deploying a bundle using maas as provider and I can only get to my LXD containers from the maas network. When on an external network, I have to go first to a server on the maas server with the right public key to be able to ssh into the LXD container.
This breaks our automated log collection for LXD containers.
This is from juju show-status:
=======
[Units]
ID WORKLOAD-STATUS JUJU-STATUS VERSION MACHINE PORTS PUBLIC-ADDRESS MESSAGE
ceilometer/0 waiting idle 2.0-beta6 4/lxd/0 8777/tcp 10.244.192.237 Incomplete relations: identity
ceph/0 active idle 2.0-beta6 0 10.244.192.199 Unit is ready and clustered
cinder/0 waiting idle 2.0-beta6 1 10.244.192.212 Incomplete relations: identity, database
glance/0 waiting idle 2.0-beta6 1/lxd/0 9292/tcp 10.244.192.223 Incomplete relations: identity, database
heat/0 waiting idle 2.0-beta6 1/lxd/1 8000/tcp,8004/tcp 10.244.192.229 Incomplete relations: identity, database
keystone/0 waiting idle 2.0-beta6 2/lxd/0 10.244.192.219 Incomplete relations: database
mongodb/0 unknown idle 2.0-beta6 3/lxd/0 27017/tcp,
mysql/0 error idle 2.0-beta6 0/lxd/0 3306/tcp 10.244.192.217 hook failed: "shared-
neutron-api/0 waiting idle 2.0-beta6 4/lxd/1 9696/tcp 10.244.192.221 Incomplete relations: identity, database
neutron-gateway/0 active idle 2.0-beta6 2 10.244.192.201 Unit is ready
nova-cloud-
nova-compute/0 error idle 2.0-beta6 4 10.244.192.214 hook failed: "install"
openstack-
rabbitmq-server/0 active idle 2.0-beta6 1/lxd/2 5672/tcp 10.244.192.233 Unit is ready
swift-proxy/0 blocked idle 2.0-beta6 2/lxd/1 8080/tcp 10.244.192.225 Not enough storage zones for minimum replicas
swift-storage/0 error idle 2.0-beta6 4/lxd/3 10.244.192.235 hook failed: "swift-
=======
If I try to ping all these addresses from my system:
=======
jenkins@
PING 10.244.192.237 (10.244.192.237) 56(84) bytes of data.
--- 10.244.192.237 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.199 (10.244.192.199) 56(84) bytes of data.
64 bytes from 10.244.192.199: icmp_seq=1 ttl=60 time=1.93 ms
--- 10.244.192.199 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.937/1.
PING 10.244.192.212 (10.244.192.212) 56(84) bytes of data.
64 bytes from 10.244.192.212: icmp_seq=1 ttl=60 time=1.03 ms
--- 10.244.192.212 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.030/1.
PING 10.244.192.223 (10.244.192.223) 56(84) bytes of data.
--- 10.244.192.223 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.229 (10.244.192.229) 56(84) bytes of data.
--- 10.244.192.229 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.219 (10.244.192.219) 56(84) bytes of data.
--- 10.244.192.219 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.231 (10.244.192.231) 56(84) bytes of data.
--- 10.244.192.231 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.217 (10.244.192.217) 56(84) bytes of data.
--- 10.244.192.217 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.221 (10.244.192.221) 56(84) bytes of data.
--- 10.244.192.221 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.201 (10.244.192.201) 56(84) bytes of data.
64 bytes from 10.244.192.201: icmp_seq=1 ttl=60 time=1.35 ms
--- 10.244.192.201 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.351/1.
PING 10.244.192.210 (10.244.192.210) 56(84) bytes of data.
64 bytes from 10.244.192.210: icmp_seq=1 ttl=60 time=1.04 ms
--- 10.244.192.210 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.046/1.
PING 10.244.192.214 (10.244.192.214) 56(84) bytes of data.
64 bytes from 10.244.192.214: icmp_seq=1 ttl=60 time=1.11 ms
--- 10.244.192.214 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.110/1.
PING 10.244.192.227 (10.244.192.227) 56(84) bytes of data.
--- 10.244.192.227 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.233 (10.244.192.233) 56(84) bytes of data.
--- 10.244.192.233 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.225 (10.244.192.225) 56(84) bytes of data.
--- 10.244.192.225 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
PING 10.244.192.235 (10.244.192.235) 56(84) bytes of data.
--- 10.244.192.235 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
=======
If I do the ping on system on the maas network:
=======
jenkins@
Warning: Permanently added '10.244.192.199' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 3.13.0-85-generic x86_64)
* Documentation: https:/
System information as of Fri Apr 29 12:44:18 UTC 2016
System load: 0.0 Users logged in: 0
Usage of /: 9.8% of 49.08GB IP address for br-eth0: 10.244.192.199
Memory usage: 24% IP address for br-eth1: 10.244.226.74
Swap usage: 0% IP address for lxcbr0: 10.0.3.1
Processes: 129 IP address for lxdbr0: 10.0.4.1
Graph this data and manage this system at:
https:/
Get cloud support with Ubuntu Advantage Cloud Guest:
http://
0 packages can be updated.
0 updates are security updates.
Last login: Fri Apr 29 12:44:20 2016 from 10.245.162.103
ubuntu@
PING 10.244.192.237 (10.244.192.237) 56(84) bytes of data.
64 bytes from 10.244.192.237: icmp_seq=1 ttl=64 time=0.669 ms
--- 10.244.192.237 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.669/0.
PING 10.244.192.199 (10.244.192.199) 56(84) bytes of data.
64 bytes from 10.244.192.199: icmp_seq=1 ttl=64 time=0.035 ms
--- 10.244.192.199 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.035/0.
PING 10.244.192.212 (10.244.192.212) 56(84) bytes of data.
64 bytes from 10.244.192.212: icmp_seq=1 ttl=64 time=0.582 ms
--- 10.244.192.212 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.582/0.
PING 10.244.192.223 (10.244.192.223) 56(84) bytes of data.
64 bytes from 10.244.192.223: icmp_seq=1 ttl=64 time=0.617 ms
--- 10.244.192.223 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.617/0.
PING 10.244.192.229 (10.244.192.229) 56(84) bytes of data.
64 bytes from 10.244.192.229: icmp_seq=1 ttl=64 time=0.459 ms
--- 10.244.192.229 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.459/0.
PING 10.244.192.219 (10.244.192.219) 56(84) bytes of data.
64 bytes from 10.244.192.219: icmp_seq=1 ttl=64 time=0.556 ms
--- 10.244.192.219 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.556/0.
PING 10.244.192.231 (10.244.192.231) 56(84) bytes of data.
64 bytes from 10.244.192.231: icmp_seq=1 ttl=64 time=0.511 ms
--- 10.244.192.231 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.511/0.
PING 10.244.192.217 (10.244.192.217) 56(84) bytes of data.
64 bytes from 10.244.192.217: icmp_seq=1 ttl=64 time=0.527 ms
--- 10.244.192.217 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.527/0.
PING 10.244.192.221 (10.244.192.221) 56(84) bytes of data.
64 bytes from 10.244.192.221: icmp_seq=1 ttl=64 time=0.561 ms
--- 10.244.192.221 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.561/0.
PING 10.244.192.201 (10.244.192.201) 56(84) bytes of data.
64 bytes from 10.244.192.201: icmp_seq=1 ttl=64 time=0.369 ms
--- 10.244.192.201 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.369/0.
PING 10.244.192.210 (10.244.192.210) 56(84) bytes of data.
64 bytes from 10.244.192.210: icmp_seq=1 ttl=64 time=0.651 ms
--- 10.244.192.210 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.651/0.
PING 10.244.192.214 (10.244.192.214) 56(84) bytes of data.
64 bytes from 10.244.192.214: icmp_seq=1 ttl=64 time=0.365 ms
--- 10.244.192.214 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.365/0.
PING 10.244.192.227 (10.244.192.227) 56(84) bytes of data.
64 bytes from 10.244.192.227: icmp_seq=1 ttl=64 time=0.504 ms
--- 10.244.192.227 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.504/0.
PING 10.244.192.233 (10.244.192.233) 56(84) bytes of data.
64 bytes from 10.244.192.233: icmp_seq=1 ttl=64 time=1.04 ms
--- 10.244.192.233 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.048/1.
PING 10.244.192.225 (10.244.192.225) 56(84) bytes of data.
64 bytes from 10.244.192.225: icmp_seq=1 ttl=64 time=0.420 ms
--- 10.244.192.225 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.420/0.
PING 10.244.192.235 (10.244.192.235) 56(84) bytes of data.
64 bytes from 10.244.192.235: icmp_seq=1 ttl=64 time=0.573 ms
--- 10.244.192.235 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.573/0.
ubuntu@
=======
I am attaching some net config file and route info for one container. I am also including route info for host in the attachment.
These are the pkg versions for lxd/lxc:
ubuntu@
ii liblxc1 1.1.5-0ubuntu3~
ii lxc 1.1.5-0ubuntu3~
ii lxc-templates 1.1.5-0ubuntu3~
ii lxcfs 2.0.0-0ubuntu2~
ii python3-lxc 1.1.5-0ubuntu3~
ubuntu@
ii lxd 2.0.0-0ubuntu1~
ii lxd-client 2.0.0-0ubuntu1~
and maas version:
ubuntu@
ii maas 1.9.1+bzr4543-
ii maas-cli 1.9.1+bzr4543-
ii maas-cluster-
ii maas-common 1.9.1+bzr4543-
ii maas-dhcp 1.9.1+bzr4543-
ii maas-dns 1.9.1+bzr4543-
ii maas-proxy 1.9.1+bzr4543-
ii maas-region-
ii maas-region-
ii python-django-maas 1.9.1+bzr4543-
ii python-maas-client 1.9.1+bzr4543-
ii python-
and juju:
jenkins@
ii juju 2.0-beta6-
ii juju-2.0 2.0-beta6-
Changed in juju-core: | |
milestone: | none → 2.0-beta7 |
Changed in juju-core: | |
milestone: | 2.0-beta7 → 2.0-beta8 |
Changed in juju-core: | |
milestone: | 2.0-beta8 → 2.0-beta9 |
Changed in juju-core: | |
assignee: | nobody → Dimiter Naydenov (dimitern) |
Changed in juju-core: | |
status: | Triaged → In Progress |
Changed in juju-core: | |
status: | In Progress → Fix Committed |
tags: | removed: blocker |
Changed in juju-core: | |
status: | Fix Committed → Fix Released |
affects: | juju-core → juju |
Changed in juju: | |
milestone: | 2.0-beta9 → none |
milestone: | none → 2.0-beta9 |
Changed in juju-core: | |
importance: | Undecided → High |
status: | New → Won't Fix |
There seems to be additional networking issue which could be related to this. I am including here unless it can be determined that it is a totally separate issue.
I did an apt-get update and it was hanging for a while at:
0% [Connecting to archive.ubuntu.com (91.189.92.201)] [Connecting to security.u
before it eventually completed.
But yesterday, it had timed out during deployment:
2016-04-28 16:43:44 INFO install Reading package lists... Dpkg::Options: :=--force- confold' ]"] Dpkg::Options: :=--force- confold' ] egenix- mxdatetime mysql-server-5.1 mysql-server python-mysqldb-dbg leadership tracker.go:182 mysql/0 will renew mysql leadership at 2016-04-28 16:44:22.964080373 +0000 UTC leadership tracker.go:182 mysql/0 will renew mysql leadership at 2016-04-28 16:44:52.964286114 +0000 UTC leadership tracker.go:182 mysql/0 will renew mysql leadership at 2016-04-28 16:45:22.964495714 +0000 UTC leadership tracker.go:182 mysql/0 will renew mysql leadership at 2016-04-28 16:45:52.964732572 +0000 UTC archive. ubuntu. com/ubuntu/ trusty-updates/main mysql-common all 5.5.49- 0ubuntu0. 14.04.1 ubuntu. com:80 (91.189.92.201), connection timed out archive. ubuntu. com/ubuntu/ trusty/main debconf-utils all 1.5.51ubuntu2 ubuntu. com:http:
2016-04-28 16:43:45 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Installing ['debconf-utils', 'python-mysqldb', 'uuid', 'pwgen', 'dnsutils'] with options: ['--option=
2016-04-28 16:43:45 INFO juju-log Installing ['debconf-utils', 'python-mysqldb', 'uuid', 'pwgen', 'dnsutils'] with options: ['--option=
2016-04-28 16:43:45 INFO install Reading package lists...
2016-04-28 16:43:45 INFO install Building dependency tree...
2016-04-28 16:43:45 INFO install Reading state information...
2016-04-28 16:43:45 INFO install dnsutils is already the newest version.
2016-04-28 16:43:45 INFO install The following packages were automatically installed and are no longer required:
2016-04-28 16:43:45 INFO install libfreetype6 os-prober
2016-04-28 16:43:45 INFO install Use 'apt-get autoremove' to remove them.
2016-04-28 16:43:45 INFO install The following extra packages will be installed:
2016-04-28 16:43:45 INFO install libmysqlclient18 libossp-uuid16 mysql-common
2016-04-28 16:43:45 INFO install Suggested packages:
2016-04-28 16:43:45 INFO install python-
2016-04-28 16:43:45 INFO install The following NEW packages will be installed:
2016-04-28 16:43:45 INFO install debconf-utils libmysqlclient18 libossp-uuid16 mysql-common pwgen
2016-04-28 16:43:45 INFO install python-mysqldb uuid
2016-04-28 16:43:52 INFO juju.worker.
2016-04-28 16:44:22 INFO juju.worker.
2016-04-28 16:44:52 INFO juju.worker.
2016-04-28 16:45:22 INFO juju.worker.
2016-04-28 16:45:46 INFO install 0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
2016-04-28 16:45:46 INFO install Need to get 780 kB of archives.
2016-04-28 16:45:46 INFO install After this operation, 4266 kB of additional disk space will be used.
2016-04-28 16:45:46 INFO install Err http://
2016-04-28 16:45:46 INFO install Could not connect to archive.
2016-04-28 16:45:46 INFO install Err http://
2016-04-28 16:45:46 INFO install Unable to connect to archive.
2016-04-28 16:45:46 INFO...