What I've forgot to say ..this is problem for connections to DB which are
ESTABLISHED in DB pool

Dne po 1. 3. 2021 12:20 uživatel Mark Goddard <email address hidden>
napsal:

> Thinking about the GARP. If the NIC was bounced, then it might not see
> the GARP from the new master. There do seem to be options to tune GARP
> transmission: https://serverfault.com/questions/821809/keepalived-send-
> gratuitous-arp-periodically
> <https://serverfault.com/questions/821809/keepalived-send-gratuitous-arp-periodically>
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1917068
>
> Title:
>   Connections to DB are refusing to die after VIP is switched
>
> Status in kolla-ansible:
>   In Progress
> Status in kolla-ansible train series:
>   New
> Status in kolla-ansible ussuri series:
>   New
> Status in kolla-ansible victoria series:
>   New
> Status in kolla-ansible wallaby series:
>   In Progress
>
> Bug description:
>   Hi,
>
>   On production kolla-ansible installed ENV we found strange bahaviour
> when switching VIP between controllers (under load).
>   When VIP is switched from master to backup keepalived, connections to DB
> are dead on host where VIP was before switch (keystone wsgi workers are all
> busy and waiting for DB reply).
>
>
>   Test env:
>   - 2 Controllers - Haproxy, keepalived, OS services, DB ..etc
>   - 2 Computes
>
>   How to reproduce:
>
>   1. Generate as big traffic as you can to replicate issue (curl  token
> issue to keystone VIP:5000)
>   2. Check logs for keystone (there will be big amount of 201 on both
> controllers)
>   2. Restart keepalived OR restart networking OR ifup/ifdown interface on
> current keepalived master
>      (VIP will be switched to secondary host)
>   3. Check logs for keystone
>   4. You can see that access log for keystone is freezed (on host where
> VIP was before), after while there will be 503,504
>
>   Why this is happening ?
>
>   Normally when master keepalived is not reachable, secondary keepalived
>   take VIP and send GARP to network, and all clients will refresh ARP
>   table, so everything should work.
>
>   Problem is that wsgi processes has connectionPool to DB and these
>   connections are dead after switch, they don't know that ARP changed
>   (probably host refused GARP because there is very tiny window when VIP
>   was assigned to him).
>
>   So, wsgi processes are trying to write to filedescriptor/socket for DB
>   connection, but waiting for reply infinite. Simply said these
>   connection are totally dead, and app layer can't fix it, because app
>   layer (oslo.db/sqlalchemy) don't know it's is broken.
>
>   Above problem is solved itself after some time -> this time depends on
>   user's kernel option net.ipv4.tcp_retries2 which is saying how many
>   retries are sent for this TCP connection before kernel will kill it.
>   In my case it was around 930-940 seconds every time I tried it
>   (default value of net.ipv4.tcp_retries2=15). Of course retransmission
>   will not work as VIP is gone and hosted by another host/mac.
>
>   Decrease tcp_retries2 to 1 fixed issue immediately.
>
>   Here is detailed article about tcp socket which are refusing to die ->
>   https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/
>
>   RedHat is also suggesting to tune this kernel option for HA solutions
>   as it is noted here -> https://access.redhat.com/solutions/726753
>
>   "In a High Availability (HA) situation consider decreasing the setting
>   to 3." << From RedHat
>
>
>   Here is also video of issue (left controller0, right contoller1, bottom
> logs, middle VIP monitor switch)
>
>   https://download.kevko.ultimum.cloud/video_debug.mp4
>
>   I will provide fix and push for review.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/kolla-ansible/+bug/1917068/+subscriptions
>