No connectivity from SNAT to FIP associated with LBaaS

[Varun Lodaya]

We recently upgraded Contrail controller from 2.21.1-15 to 2.21.2-33. vRouters are still on 2.21.1-15. Since the upgrade we've been seeing issues in connectivity between SNAT and LBaaS FIP.
After some debugging with contrail engineers, the issue seems to be discrepancy between L2 and L3 routes from SNAT to LBaaS.
1 of them points to Active HAProxy instance while the other points to standby HAProxy instance.

Following data proves that fact:

root@b0e003ash2019:/home/varun_lodaya# rt --dump 16 | grep 100.73.69.169
100.73.69.169/32 32 LP 32 31 2:1f:64:c8:2c:da(104264)
root@b0e003ash2019:/home/varun_lodaya# rt --dump 16 --famil bridge | grep 2:1f:64:c8:2c:da
104264 2:1f:64:c8:2c:da LDf 23 444
root@b0e003ash2019:/home/varun_lodaya# nh --get 31
Id:31 Type:Tunnel Fmly: AF_INET Flags:Valid, MPLSoUDP, Rid:0 Ref_cnt:36 Vrf:0
              Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:7c 69 f6 e0 95 ad a0 36 9f 3a ac 9a 08 00
              Vrf:0 Sip:10.50.146.181 Dip:100.72.154.89

root@b0e003ash2019:/home/varun_lodaya# nh --get 23
Id:23 Type:Tunnel Fmly: AF_INET Flags:Valid, MPLSoGRE, Rid:0 Ref_cnt:9 Vrf:0
              Oif:0 Len:14 Flags Valid, MPLSoGRE, Data:7c 69 f6 e0 95 ad a0 36 9f 3a ac 9a 08 00
              Vrf:0 Sip:10.50.146.181 Dip:10.50.128.6

L3 DIP seems to be 100.72.154.89 while L2 DIP seems to be 10.50.128.6