Comment 8 for bug 1917068

Hi,

Well yes, thank you that you replicated same. I was investigating this
issue and how to fix longer then week :)

I already proposed patch, but I am planning change my review to have this
option configurable .. today or tomorrow.

Just want to say, for example k8s is very sensitive when keystone is
stucked ...

Michal

Dne po 1. 3. 2021 23:25 uživatel chalansonnet <email address hidden>
napsal:

> Hello,
>
> Just tried on my environment :
> Centos7.8 RDO deploy with Kolla-Ansible Stein release
> 2 network node with Haproxy and keepalived
> 3 Ctrl nodes with Keystone
> 3 BDD node with Rabbit & Galera
>
> net.ipv4.tcp_retries2 default value :15 retries
>
> Steps i followed :
> Generate some loops requests to keystone .
> Log onto the master keepalived with Vip public and private
> Shutoff the keepalived container
> => VIP was recreated almost instant on the other network controler
>
>
> Logs on Keystone
> Lost access to the Vip Database <= maybe it is different configuration
> from you, but all services are connected under the Mariadb VIP
> Requests to keystone was stuck during 180sec
>
> Second test
> net.ipv4.tcp_retries2 set to 10
> Request to keystone was during 120sec
>
> Third test
> net.ipv4.tcp_retries2 set to 5
> Request to keystone was during 60sec
>
> So, you are right , we can tune the failover of the VIP Haproxy with this
> tuning !!
> I will do another tests , for me 180sec to failover was acceptable.
>
> Greeting
> Stephane Chalansonnet
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1917068
>
> Title:
> Connections to DB are refusing to die after VIP is switched
>
> Status in kolla-ansible:
> In Progress
> Status in kolla-ansible train series:
> New
> Status in kolla-ansible ussuri series:
> New
> Status in kolla-ansible victoria series:
> New
> Status in kolla-ansible wallaby series:
> In Progress
>
> Bug description:
> Hi,
>
> On production kolla-ansible installed ENV we found strange bahaviour
> when switching VIP between controllers (under load).
> When VIP is switched from master to backup keepalived, connections to DB
> are dead on host where VIP was before switch (keystone wsgi workers are all
> busy and waiting for DB reply).
>
>
> Test env:
> - 2 Controllers - Haproxy, keepalived, OS services, DB ..etc
> - 2 Computes
>
> How to reproduce:
>
> 1. Generate as big traffic as you can to replicate issue (curl token
> issue to keystone VIP:5000)
> 2. Check logs for keystone (there will be big amount of 201 on both
> controllers)
> 2. Restart keepalived OR restart networking OR ifup/ifdown interface on
> current keepalived master
> (VIP will be switched to secondary host)
> 3. Check logs for keystone
> 4. You can see that access log for keystone is freezed (on host where
> VIP was before), after while there will be 503,504
>
> Why this is happening ?
>
> Normally when master keepalived is not reachable, secondary keepalived
> take VIP and send GARP to network, and all clients will refresh ARP
> table, so everything should work.
>
> Problem is that wsgi processes has connectionPool to DB and these
> connections are dead after switch, they don't know that ARP changed
> (probably host refused GARP because there is very tiny window when VIP
> was assigned to him).
>
> So, wsgi processes are trying to write to filedescriptor/socket for DB
> connection, but waiting for reply infinite. Simply said these
> connection are totally dead, and app layer can't fix it, because app
> layer (oslo.db/sqlalchemy) don't know it's is broken.
>
> Above problem is solved itself after some time -> this time depends on
> user's kernel option net.ipv4.tcp_retries2 which is saying how many
> retries are sent for this TCP connection before kernel will kill it.
> In my case it was around 930-940 seconds every time I tried it
> (default value of net.ipv4.tcp_retries2=15). Of course retransmission
> will not work as VIP is gone and hosted by another host/mac.
>
> Decrease tcp_retries2 to 1 fixed issue immediately.
>
> Here is detailed article about tcp socket which are refusing to die ->
> https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/
>
> RedHat is also suggesting to tune this kernel option for HA solutions
> as it is noted here -> https://access.redhat.com/solutions/726753
>
> "In a High Availability (HA) situation consider decreasing the setting
> to 3." << From RedHat
>
>
> Here is also video of issue (left controller0, right contoller1, bottom
> logs, middle VIP monitor switch)
>
> https://download.kevko.ultimum.cloud/video_debug.mp4
>
> I will provide fix and push for review.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/kolla-ansible/+bug/1917068/+subscriptions
>