Changing ports for backend, haproxy reload fails to pick up

[Haw Loeung]

Hi,

When changing backends to use different ports, or enable "backend-tls", haproxy reload fails to pick up the change. It seems to do with loading backends from the saved state file (/run/haproxy/saved-server-state).

We should figure out how to detect this and work around it (may also be a bug opened upstream).

[Haw Loeung (hloeung)]

One idea floating around is to parse the state file and look for inconsistencies then exclude those and re-write it. It's a text file:

| $ cat /run/haproxy/saved-server-state
| 1
| # be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord srv_use_ssl srv_check_port srv_check_addr srv_agent_addr srv_agent_port
| 34 backend-cached-mysite-com 1 server_1 127.0.0.1 2 0 1 1 175487 15 3 61 6 0 0 0 - 6127 - 0 0 - - 0
| 35 backend-cached-mysite-2-com 1 server_1 127.0.0.1 2 0 1 1 175487 15 3 61 6 0 0 0 - 6128 - 0 0 - - 0
| 36 backend-cached-mysite-3-com 1 server_1 127.0.0.1 2 0 1 1 175487 15 3 61 6 0 0 0 - 6130 - 0 0 - - 0
| ...

[Haw Loeung (hloeung)]

The related upstream bug looks to be this - https://github.com/haproxy/haproxy/issues/2103

Also of interest is work to redesign that area - https://github.com/haproxy/haproxy/issues/953

[Romain Couturat (romaincout)]

had the issue today,
adding some (already encountered) details :
a full restart (`sudo systemctl stop haproxy ; sleep 1; sudo systemctl start haproxy`) doesn't help,
but deleting the state file + a "simple" restart does :
`rm /run/haproxy/saved-server-state ; systemctl restart haproxy`

[Haw Loeung (hloeung)]

Re-opening. Not happy with this solution as it causes downtime for other sites hosted on the same content cache servers. Also, it doesn't solve the issue with backend IPs changing as we've seen.

Instead, I think a better solution is to parse out the saved state file and remove just specific sites from there if there are changes. That way, a HAProxy reload would only cause downtime for that site with the others seemlessly reloaded.

[Haw Loeung (hloeung)]

Filed this upstream - https://github.com/haproxy/haproxy/issues/2506