aws containers are broken in 1.23

[Curtis Hovey]

As of Commit 2e07936c in 1.23, the aws-deployer-bundle test fails. Machine 2 cannot create the containers for the two app servers which are behind HAProxy on the machine.

machines:
  "0":
    agent-state: started
    agent-version: 1.23-beta4
    dns-name: 52.5.226.249
    instance-id: i-470c67bb
    instance-state: running
    series: trusty
    hardware: arch=amd64 cpu-cores=1 cpu-power=300 mem=3840M root-disk=8192M availability-zone=us-east-1a
    state-server-member-status: has-vote
  "1":
    agent-state: started
    agent-version: 1.23-beta4
    dns-name: 52.4.228.68
    instance-id: i-b40c5a63
    instance-state: running
    series: trusty
    hardware: arch=amd64 cpu-cores=1 cpu-power=300 mem=3840M root-disk=8192M availability-zone=us-east-1c
  "2":
    agent-state: started
    agent-version: 1.23-beta4
    dns-name: 52.5.201.139
    instance-id: i-e9983b14
    instance-state: running
    series: trusty
    containers:
      2/lxc/0:
        agent-state-info: 'failed to retrieve the template to clone: template container
          "juju-trusty-lxc-template" did not stop'
        instance-id: pending
        series: trusty
      2/lxc/1:
        agent-state-info: 'lxc container cloning failed: cannot clone a running container'
        instance-id: pending
        series: trusty
    hardware: arch=amd64 cpu-cores=1 cpu-power=300 mem=3840M root-disk=8192M availability-zone=us-east-1d

Commit 2e07936c changes tghe SNAT rules to revert to the previous behaviour of container address per bug 1441811. The annoucement about container addressability explained that the feature was only available for MAAS and AWS, implying there were other changes made that also need to be reverted.

This issue looks like bug 1441319, but we know exactly which commit broke it. I captured the container log from the machine before it was destroyed (https://launchpadlibrarian.net/202761839/container.log).

Using the bundle found at
    http://bazaar.launchpad.net/~juju-qa/juju-ci-tools/repository/view/head:/bundles.yaml
you can run the same test as CI
    juju bootstrap
    juju --show-log deployer --debug --deploy-delay 10 --config bundles.yaml

This works on 1.22.1 and 1.23-beta4 (cut form cherylj's commit).
Starting with commit 2e07936c (which thinks it is 1.23-beta4, but is not) deployments with containers in aws will fail.

[Dimiter Naydenov (dimitern)]

Please, provide logs at TRACE level from the machine in question (hosting the containers failed to start), as well as the console.log, lxc.conf, and cloud-init-output.log from the container. Basically the contents of /var/log/juju/machine-X.log, /var/lib/juju/containers/*/*.log, /var/lib/lxc/*/config, /var/lib/lxc/*/rootfs/var/log/cloud-init*.log. The given container.log is not useful.

[James Tunnicliffe (dooferlad)]

I started an environment without a container cloning, which didn't work. This seems to be because we are getting anIPv6 address for archive.ubuntu.com and EC2 doesn't support IPv6:

From cloud-init-output.log

Err http://archive.ubuntu.com/ubuntu/ trusty/main libaio1 amd64 0.3.109-4
  Cannot initiate the connection to archive.ubuntu.com:80 (2001:67c:1360:8c01::19). - connect (101: Network is unreachable) [IP: 2001:67c:1360:8c01::19 80]

root@juju-machine-0-lxc-0:/etc# host archive.ubuntu.com
archive.ubuntu.com has address 91.189.92.201
archive.ubuntu.com has address 91.189.88.149
archive.ubuntu.com has address 91.189.91.13
archive.ubuntu.com has address 91.189.91.14
archive.ubuntu.com has address 91.189.91.15
archive.ubuntu.com has address 91.189.91.23
archive.ubuntu.com has address 91.189.91.24
archive.ubuntu.com has address 91.189.92.200
archive.ubuntu.com has IPv6 address 2001:67c:1360:8c01::19
archive.ubuntu.com has IPv6 address 2001:67c:1360:8c01::18

[James Tunnicliffe (dooferlad)]

Disabling IPv6 doesn't help much, but it does stop us trying to use it:

root@juju-machine-0-lxc-0:/etc# echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6
root@juju-machine-0-lxc-0:/etc# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2014ms

root@juju-machine-0-lxc-0:/etc# host archive.ubuntu.com
archive.ubuntu.com has address 91.189.91.15
archive.ubuntu.com has address 91.189.91.23
archive.ubuntu.com has address 91.189.91.24
archive.ubuntu.com has address 91.189.92.200
archive.ubuntu.com has address 91.189.92.201
archive.ubuntu.com has address 91.189.88.149
archive.ubuntu.com has address 91.189.91.13
archive.ubuntu.com has address 91.189.91.14
archive.ubuntu.com has IPv6 address 2001:67c:1360:8c01::18
archive.ubuntu.com has IPv6 address 2001:67c:1360:8c01::19
root@juju-machine-0-lxc-0:/etc# ifconfig
eth0 Link encap:Ethernet HWaddr 00:16:3e:4d:95:16
          inet addr:172.31.1.157 Bcast:255.255.255.255 Mask:255.255.255.255
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:5402 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3754 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:11164918 (11.1 MB) TX bytes:722789 (722.7 KB)

So the first thing to do is not hand out IPv6 addresses over DHCP.

[James Tunnicliffe (dooferlad)]

Ah, no conntrack rule for iptables. Surely we need one?! Some form of nat would be good...

root@ip-172-31-2-128:/var/lib/lxc/juju-machine-0-lxc-0/rootfs/var/log# iptables-save
# Generated by iptables-save v1.4.21 on Mon Apr 13 14:34:41 2015
*mangle
:PREROUTING ACCEPT [205816:779016340]
:INPUT ACCEPT [205448:778989482]
:FORWARD ACCEPT [368:26858]
:OUTPUT ACCEPT [150587:599629711]
:POSTROUTING ACCEPT [150955:599656569]
-A POSTROUTING -o lxcbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
COMMIT
# Completed on Mon Apr 13 14:34:41 2015
# Generated by iptables-save v1.4.21 on Mon Apr 13 14:34:41 2015
*nat
:PREROUTING ACCEPT [100:6355]
:INPUT ACCEPT [24:1420]
:OUTPUT ACCEPT [202:13968]
:POSTROUTING ACCEPT [275:18699]
-A POSTROUTING -s 10.0.3.0/24 ! -d 10.0.3.0/24 -j MASQUERADE
COMMIT
# Completed on Mon Apr 13 14:34:41 2015
# Generated by iptables-save v1.4.21 on Mon Apr 13 14:34:41 2015
*filter
:INPUT ACCEPT [205059:778815960]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [150200:599483423]
-A INPUT -i lxcbr0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i lxcbr0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i lxcbr0 -p tcp -m tcp --dport 67 -j ACCEPT
-A INPUT -i lxcbr0 -p udp -m udp --dport 67 -j ACCEPT
-A FORWARD -s 172.31.0.0/20 -i lxcbr0 -j ACCEPT
-A FORWARD -d 172.31.0.0/20 -o lxcbr0 -j ACCEPT
-A FORWARD -o lxcbr0 -j ACCEPT
-A FORWARD -i lxcbr0 -j ACCEPT
COMMIT
# Completed on Mon Apr 13 14:34:41 2015

[James Tunnicliffe (dooferlad)]

Expect that this:
"iptablesSNAT": {
- "nat",
- "POSTROUTING",
- "-o {{.HostIF}} -j SNAT --to-source {{.HostIP}}",
- },

Should be:
"iptablesSNAT": {
- "nat",
- "POSTROUTING",
- "-o {{.HostIF}} -j SNAT --to-source {{.ContainerIP}}",
- },

[Dimiter Naydenov (dimitern)]

We did find a working solution:
-A POSTROUTING -o {{.HostIF}} -d {{.ContainerCIDR}} -j RETURN # ensures cloud-internal traffic originates from the proper IP
-A POSTROUTING -o {{.HostIF}} -j SNAT --to-source {{.HostIP}} # ensures egress for the container works

ContainerCIDR = CIDR for the host subnet (e.g. 172.31.0.0/20 for AWS)

Tested on AWS and is being tested on MAAS - a fix should be proposed soon

[James Tunnicliffe (dooferlad)]

We have a fix reviewed and providing all goes well in CI land will land shortly.

Needed to have the SNAT rule in for all traffic leaving the VPC. This isn't required for non-EC2 and it didn't seem to be anything to do with the source/destination check (turning it off didn't help). I have tested LXC <--> another physical machine on both EC2 and MAAS within the same network and that traffic has the container IP address as the source address. LXC <--> WWW works too.

[Dimiter Naydenov (dimitern)]

Additionally, we discovered MAAS shouldn't have any SNAT rule, but EC2 does. So I've proposed a follow-up hotfix - https://github.com/juju/juju/pull/2076

Landing to unblock 1.23.0 release. Will do a better tested fix for master.

[Dimiter Naydenov (dimitern)]

So once Michael's work around forward porting the feature-flagged addressable containers lands on 1.24, this can be deferred past the first dev 1.24 release, because it will only happen in feature-flagged environments (without the flag containers won't be addressable anyway).

[Curtis Hovey (sinzui)]

The fix for 1441826 appears to work. while all deployer and quickstart jobs were broken in master and 1.24 for weeks, 6 of the 8 pass. The aws-deployer-bundle job still fails though. The issue in the lxc containers. We know that 1.23 passed with the fix associated with the bug, and that it was not merged into master, now 1.124 was forked. If the failure in

    http://reports.vapour.ws/releases/2576/job/aws-deployer-bundle/attempt/256

is not related to this issue, we need to report a separate blocker+ci bug.

[Dimiter Naydenov (dimitern)]

This should be resolved once https://github.com/juju/juju/pull/2173 lands, which should be today.

By "resolved", I mean it will work just like it did in 1.22.x (containers won't be addressable on AWS). We can have a separate aws-deployer-bundle-ca-ff job that exercises the same test but with JUJU_DEV_FEATUREF_FLAGS=address-allocation at bootstrap.

[Dimiter Naydenov (dimitern)]

I've just discovered the NAT fix done for 1.23 (with https://github.com/juju/juju/pull/2071/) has not landed on 1.24 (or master). It's easy enough to fix this, I'll take care of it. What I said in comment #11 still stands though - once PR #2173 lands (on master, it should've been proposed against 1.24 first FWIW) AWS containers won't be addressable by default.

I'll take PR #2173 and backport it to 1.24 and land that before #2173 lands.

[Dimiter Naydenov (dimitern)]

Fix for 1.24 proposed with https://github.com/juju/juju/pull/2187

[Dimiter Naydenov (dimitern)]

I'm assigning James to forward-port this to master. James, please do also a live tests on MAAS and EC2 because even though 1.24 and master are still quite similar, I really don't want to deal with new regressions :)

[Dimiter Naydenov (dimitern)]

During live testing this I discovered yet another thing was missing from 1.23 - https://github.com/juju/juju/pull/2076, so I'm currently live testing the forward port of that on top of #2187 for 1.24 on MAAS and EC2.

[Dimiter Naydenov (dimitern)]

Correction, my previous comment was more about bug 1443942, so the fix that landed is correct for this bug. However, the forward-port of #2076 to 1.24 is just one step towards fixing #1443942 (there are no unit tests, just live tested and does not handle the EC2 VPC subnets specifically).