LXC containers getting HA VIP addresses after reboot

[Peter Sabaini]

Due to network driver issues a juju-db node in a HA configuration had to be rebooted. After the node came back up, several units had their public addresses set incorrectly. Those units were part of a corosync HA cluster. For some of the units, the public address has been set to the corosync VIP that should normally float between nodes, instead of their normal machine address. This prevented corosync from starting up and therefore made the service inaccessible. Rebooting the containers set the unit addresses back to their correct value.

The behaviour is similar to the one described in bug #1463480, where VIP addresses had been set in the course of a juju upgrade.; only here no upgrade took place, just a node reboot.

Juju version: 1.22.6-trusty-amd64

[Cheryl Jennings (cherylj)]

I don't think, for bug #1463480, that an upgrade was required to hit this problem. I think just rebooting the node was enough for the IP to switch. Going to just double check with mfoord.

[Peter Sabaini (peter-sabaini)]

I suspected that this might be related. However one notable diff I should point out: for bug #1463480 to hit, the HA VIP has to sort first, before the "regular" ie. correct addresses. In this case, the VIPs do sort _after_ the regular addresses (we've moved our default VIP block in honour of the above bug).

[Cheryl Jennings (cherylj)]

Thanks for the additional info, Peter. Can you attach the machine logs?

[Peter Sabaini (peter-sabaini)]

Here's the logs from machine-0 and machine-1 (the one that was rebooted):
https://chinstrap.canonical.com/~sabaini/2015-11-13-jujureboot-ha-issue

[Cheryl Jennings (cherylj)]

While this seems to have been fixed for the non-container case, I can still reproduce this problem with 1.25.5 and 2.0-beta7.

Steps:
- Deploy a service to a container
- Add an additional IP to eth0: sudo ip addr add 9.9.9.9 dev eth0 (it is important that the IP is lower than the current IP on eth0)
- Restart the jujud process on the container

You'll see that the IP is switched:
      0/lxc/0:
        juju-status:
          current: started
          since: 11 May 2016 11:58:28-05:00
          version: 2.0-beta7.1
        dns-name: 9.9.9.9

[John A Meinel (jameinel)]

The machine agent looks at the IP addresses on the network devices at startup. So it shouldn't require an instance reboot, just a simple restart of the jujud agent. (restart jujud-machine-X, IIRC)

I thought we would have some of the same fixes in place for containers that we had for machines, but here are my thoughts on it:
  1. Check your IP addresses more regularly, so that when you lose the VIP you notice and can remove the VIP from your list. However, their charms would have to deal with the IP address changing, which I'm not sure if they are equipped to do.
  2. Hysteresis where we remember the last address we gave and continue to give that address. This requires making sure the machine agent starts up at some point where we don't have the VIP, so that it will start reporting the non-VIP address, and then future reboots of the agent should be ok. (I thought we did this one)
  3. Space awareness, where we can notice that the VIP address is not in the space that we want the unit to be reporting from. This is potentially risky depending on how they are defining the spaces and what VIP address they are giving. (If they say the subnet is 10.0.0/24 but then give 10.0.0.254 as the VIP then we can't tell it isn't in the right subnet.) I don't know how they are picking VIP, I believe they are picking 'exclusion' ranges in MAAS, but I don't know how that overlaps with the subnets that they are defining.
  4. Stop using detected addresses entirely, in favor of Provider reported addresses. We may need to vet this a bit more, but since we now use the Device mechanism from MAAS to statically allocate an IP address, we should be able to know the IP address without waiting for the machine to come up. We were using detection to handle the fact that DHCP allocated addresses weren't known in advance. (do we have an 'ignore-machine-addresses' flag that was intended to do this, but might have a different bug in it?)

[Cheryl Jennings (cherylj)]

IIRC, the ignore-machine-addresses flag is ignored for containers (see bug 1509292).

[Dimiter Naydenov (dimitern)]

I managed to isolate the problem, and it's not due to addresses sort order. It's because the HA VIP has a "public" scope (e.g. 109.106.176.12 or 9.9.9.9 as described in comment #5), while the other addresses have "local-cloud" scope (RFC 1918 - e.g. 172.16.150.107). What you see in "dns-name" is taken from the preferred public address of the machine (returned also by 'unit-get public-address'). Addresses sort order only matters within the same scope.

To avoid hitting the described issue, I *strongly* suggest not using non-RFC1918 ("public") IPs for HA VIPs. I can confirm using the steps in comment #5 and any IP from the same subnet (sorting lower OR higher than the existing one on eth0) and restarting jujud *will not* change the "dns-name" for the machine in status.

To fix the issue properly the corosync charm can be changed to use network-get with a given binding, which is reserved for clustering/HA endpoints and *not* to use 'unit-get private-address' or 'unit-get public-address'. We should discuss the details how to do that.

[Dimiter Naydenov (dimitern)]

Not an actual fix, but at least to make the situation less surprising I'll propose a patch adding extra logging if and when the preferred private/public addresses of a machine change.

[Peter Sabaini (peter-sabaini)]

We _are_ using RFC1918 addresses for HA VIP as a rule. This should also be visible in the logs mentioned above. Note due to infra changes the log url is now https://private-fileshare.canonical.com/~sabaini/2015-11-13-jujureboot-ha-issue/

[Dimiter Naydenov (dimitern)]

Sorry Peter, it's not clear to me what ranges are you using for VIPs from the logs. I've seen both e.g. 109.106.176.12 and 172.16.150.107.

Could you describe the networks you are using please?

[Dimiter Naydenov (dimitern)]

Here's a patch that will add extra logging at INFO level whenever the preferred private or public addresses of a machine change (typically picked up after host reboots or the running jujud is restarted). It should help to track when and why the unexpected changes to HA VIPs happen.

https://github.com/juju/juju/pull/5980

This should be part of the next beta (15).