L3 HA: Unstable rescheduling time for keepalived v1.2.7

[Ann Taraday]

I have tested work of L3 HA on environment with 3 controllers and 1 compute (Kilo) with this simple scenario:
1) ping vm by floating ip
2) disable master l3-agent (which ha_state is active)
3) wait for pings to continue and another agent became active
4) check number of packages that were lost

My results are following:
1) When max_l3_agents_per_router=2, 3 to 4 packages were lost.
2) When max_l3_agents_per_router=3 or 0 (meaning the router will be scheduled on every agent), 10 to 70 packages were lost.

I should mention that in both cases there was only one ha router.

It is expected that less packages will be lost when max_l3_agents_per_router=3(0).

[Assaf Muller (amuller)]

What is your installed keepalived version? Check out https://bugs.launchpad.net/neutron/+bug/1433172.

[Assaf Muller (amuller)]

More context: That bug that I linked (Which is a bug in certain keepalived versions) only manifests if you have more than two L3 agents participating in the HA router domains, which would explain the behavior that you're seeing. If you look at /var/log/syslog or messages, if you're using a 'bad' version of keepalived, you'll see that keepalived is flapping between backup and master.

[Ann Taraday (akamyshnikova)]

I'm using Keepalived v1.2.7 (08/14,2013). I checked /var/log/syslog and I don't find any strange messages from keepalived .

[Assaf Muller (amuller)]

If you monitor /var/log/syslog for the node you're killing, and for the other two nodes, what messages are you seeing from keepalived?

[Assaf Muller (amuller)]

And how do the timestamps look like? Does it take minutes for keepalived itself to stabilize? If so I'd try with something like keepalived 1.2.13.

[Ann Taraday (akamyshnikova)]

I've updated keepalived to 1.2.19 and now L3 HA works very nice! Thanks a lot!
The only concern here is that keeplived had a lot of dependencies, so updating it can be inconvenient. Do we have a list of reliable versions of keepalived? May be I've missed this, if not it would be good to have it.

[Assaf Muller (amuller)]

As you can imagine I've been developing and testing L3 HA on Fedora/CentOS and RHEL. When I started working on, v1.2.13 was the latest, and it works well. v1.2.14 is broken, so are early versions of v1.2.15. v1.2.16+ should be good. IPv6 support works from v1.2.10+. That's all I've got :) This should probably be documented properly in the network guide...

[Ann Taraday (akamyshnikova)]

Thanks a lot!

It is good to add this in network guide, as Ubuntu(12.04;14.04;14.10;15.04) is used keepalived v1.2.7 and only in Ubuntu 15.10 keepalived v1.2.19 is going to be used.

[Akihiro Motoki (amotoki)]

Per bug discussion, it is better to add a note on appropriate keepalived versions to the networking guide.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/338129

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/338129
Committed: https://git.openstack.org/cgit/openstack/openstack-ansible-os_neutron/commit/?id=0b8721141f9b526ba4902f5cfc53f05c2fc0758e
Submitter: Jenkins
Branch: master

commit 0b8721141f9b526ba4902f5cfc53f05c2fc0758e
Author: Jean-Philippe Evrard <email address hidden>
Date: Wed Jul 6 10:10:52 2016 +0100

    Use UCA for non-OVS neutron

    This commit refactors tasks to allow the use of UCA for Linux Bridge.
    It also changes default behavior: now every neutron install will
    make use of Ubuntu Cloud Archive, unless mentionned.

    Closes-Bug: 1497272
    Closes-Bug: 1433172

    Change-Id: I4373f544eb178720f33795a71adae925a8b8cb03
    Signed-off-by: Jean-Philippe Evrard <email address hidden>

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/openstack-ansible-os_neutron 14.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/430206

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/430206
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=430909f6e15e56308c3895007e20a91f73b9412a
Submitter: Jenkins
Branch: master

commit 430909f6e15e56308c3895007e20a91f73b9412a
Author: John Davidge <email address hidden>
Date: Tue Feb 7 11:20:31 2017 +0000

    [networking] Add a note on bug in keepalived

    Describes how a bug in keepalived v1.2.15 and earlier can affect
    operation of neutron features, and recommends upgrading to a greater
    version to avoid problems.

    Change-Id: I05de49e0043347b2cfcce3af8cf68796f70334b9
    Closes-Bug: #1497272

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-manuals 15.0.0 release.

[Ihar Hrachyshka (ihar-hrachyshka)]

It's keepalived issue, and supported platforms like centos/rhel or xenial, already ship fixed packages. We also documented the issue in networking guide. There seems to be nothing we can do more on neutron side, so moving the bug to Won't Fix.