Keepalived < 2.0.x in Ubuntu 18.04 LTS not compatible with systemd-networkd

[cdmiller]

Systemd-networkd clobbers VIPs placed by other daemons on any reconfiguration triggering systemd-networkd restart (netplan apply for example). Keepalived < version 2.0.x will not restore a VIP lost in this fashion, breaking high availability on Ubuntu 18.04 LTS. A backport for keepalived >= 2.0.x should fix the issue.

[Christian Ehrhardt  (paelzer)]

FYI that is about:
https://lists.ubuntu.com/archives/ubuntu-server/2018-October/007761.html
https://lists.ubuntu.com/archives/ubuntu-server/2018-November/007784.html
https://lists.ubuntu.com/archives/ubuntu-server/2019-February/007811.html
https://lists.ubuntu.com/archives/ubuntu-server/2019-March/007819.html

[Christian Ehrhardt  (paelzer)]

(or at least related)

[Christian Ehrhardt  (paelzer)]

Adding other components involved.

[Christian Ehrhardt  (paelzer)]

Is this a dup to bug 1815101 ?

[Karl Stenerud (kstenerud)]

Also https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1810583

They all seem related.

[cdmiller (cdmiller)]

Bug 1815101 is a symptom of this bug.

I do not think bug 1810583 is related, I do not think a backport of keepalived >= 2.0.x would fix bug 1810583.

Thanks,

- cameron

[cdmiller (cdmiller)]

After reading the comments for bug 1810583 closer, it is also a symptom of this bug, and keepalived >= 2.0.x would fix it (restore the VIPs).

Thanks,

- cameron

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Lars Erik Pedersen (pedersen-larserik)]

Will anything be done here? This basically breaks keepalived for bionic. I.e, will keepalived >= 2.0.x be available for bionic?

[Bryce Harrington (bryce)]

Given SRU and backport team policies, having a newer version of keepalived in bionic seems pretty unlikely. SRU policy favors backports of specific fixes rather than entire package backports, while the backports team generally discourages backports of libraries or services since they can randomly break other software using them.

So, the most practical route forward would be to identify the patch(es) needed for fixing the particular issue at hand, and go through the regular SRU process. I am, unfortunately, completely unfamiliar with keepalived, but attached is a list of upstream comments mentioning "VIP" since the v.1.3.9 release, which I generated like this:

    git log --grep="VIP" v1.3.9.. > /tmp/vip_commits.txt

The next step would be for someone more familiar than me, to review the list and identify 1 or 2 patches worth testing. Then apply the patch to the bionic keepalived package and test for a fix. After we know what patch is needed, an SRU request can be placed to have it released for all users.

[Bryce Harrington (bryce)]

Oh, one other thing that will be necessary to file an SRU for this would be a a series of "paint by number" steps to reproduce the issue, and to verify the fix. For example, something akin to: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1810583/comments/12

[Christian Ehrhardt  (paelzer)]

Fixed in later versions, Rafael will take a HA-POV look at this after FF in Eoan.

[Wes (wes234234)]

Medium? I've had this take a few prod clusters offline at highly inconvenient times, it effectively makes keepalived on 18.04 unfit for purpose. We've had to build our own keepalived packages to keep things running - I think you need to give this a bit more priority please

[Edward Hope-Morley (hopem)]

Looks like this has been fixed in keepalived 2.x (detection of missing vip) - https://github.com/acassen/keepalived/issues/836 - but the patch is embedded with a whole load others that were merged at once so might be hard to backport.

[Rafael David Tinoco (rafaeldtinoco)]

The following 3 bugs:

https://bugs.launchpad.net/bugs/1815101
https://bugs.launchpad.net/bugs/1819074
https://bugs.launchpad.net/bugs/1810583

Have the same root cause: the fact that systemd-network messes with secondary IP addresses in NICs managed by systemd.

I'm marking all other cases as a duplicate of LP: #1815101.

TODO here is the following:

- There are mainly 2 "fixes" for this issue:

1) keepalived is able to recognize systemd-networkd changes and change cluster status in order to reconfigure managed NICs (keepalived (> 2.0.x)).

2) systemd-networkd implements a new stanza (KeepConfiguration=) to systemd service unit files in order to fix not only this behavior but all those HA related software that manages secondary IPs and/or aliases to NICs being managed by systemd-networkd.

I think the most appropriate would make sure those 2 features work in Eoan, both, together, and then make sure the SRUs are done to Disco and Bionic. One problem w/ the item (2) is that netplan will also have to support the new "KeepConfiguration=" systemd service file stanza, but, the fix (2) is more appropriate for all other HA related softwares controlling virtual IPs (CTDB, Pacemaker, and so ...).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.