How to reproduce:
On a setup with two L3 agents, create ten HA routers, the scheduler will place them on both agents, but the same agent will host the active instance of all ten routers. This defeats the idea of load sharing traffic across all L3 agents.
Solutions:
This can be solved in one of two ways:
1) Enable preemptive elections for HA routers. Keepalived enables a configuration value that causes VRRP pre-emptive elections. This way we can set a random VRRP priority for each router instance, and the elections process will guarantee a random distribution of active routers on the available agents. Preemptive elections have a major downside - If an agent that's hosting a master instance drops, the backup router will come in to play, but when the node is fixed the old master will re-assume its role. This second state transition is costly and redundant.
2) With non-preemptive elections the first router instance to come up will become the master. We can exploit this and send the notification from the server to the agents in a random order.
Why is 1 costly and redundant? A VRRP takeover isn't that bad. The other advantage is that the load will then be shared again across the agents where if you don't have preemption everything will be stuck on one router after the failure.