inconsistency between ifmap and api-server

[Senthilnathan Murugappan]

On a scale setup with 5 controllers observed that one of the ifmap servers isnt consistent.
Was doing clear and set of gateway of a snat instance.
The clear would delete the SI and its associated RIs. Observed that the same is removed on all api-servers however one of the ifmap-server still had the RoutingInstances.

We dont have any logs around the timeframe due to an issue with collector and the contrail api logfiles too were rolled over.

root@b5s29:~# grep 7fb86f6034dd_2be83366-32e3-4215-9c29-572241d12f5e /tmp/ifmap-output
          <identity name="contrail:virtual-machine-interface:default-domain:TestProject6D2DC8:default-domain__TestProject6D2DC8__snat_b243f11f-1c14-4237-b4b7-7fb86f603
4dd_2be83366-32e3-4215-9c29-572241d12f5e__1__left__2" type="other" other-type-definition="extended"/>
          <identity name="contrail:virtual-machine-interface:default-domain:TestProject6D2DC8:default-domain__TestProject6D2DC8__snat_b243f11f-1c14-4237-b4b7-7fb86f603
4dd_2be83366-32e3-4215-9c29-572241d12f5e__2__left__2" type="other" other-type-definition="extended"/>
          <identity name="contrail:service-instance:default-domain:TestProject6D2DC8:snat_b243f11f-1c14-4237-b4b7-7fb86f6034dd_2be83366-32e3-4215-9c29-572241d12f5e" type="other" other-type-definition="extended"/>
          <identity name="contrail:virtual-machine-interface:default-domain:TestProject6D2DC8:default-domain__TestProject6D2DC8__snat_b243f11f-1c14-4237-b4b7-7fb86f603
4dd_2be83366-32e3-4215-9c29-572241d12f5e__2__right__1" type="other" other-type-definition="extended"/>
          <identity name="contrail:virtual-machine-interface:default-domain:TestProject6D2DC8:default-domain__TestProject6D2DC8__snat_b243f11f-1c14-4237-b4b7-7fb86f603
4dd_2be83366-32e3-4215-9c29-572241d12f5e__1__right__1" type="other" other-type-definition="extended"/>
          <identity name="contrail:service-instance:default-domain:TestProject6D2DC8:snat_b243f11f-1c14-4237-b4b7-7fb86f6034dd_2be83366-32e3-4215-9c29-572241d12f5e" type="other" other-type-definition="extended"/>
          <identity name="contrail:virtual-machine:default-domain__TestProject6D2DC8__snat_b243f11f-1c14-4237-b4b7-7fb86f6034dd_2be83366-32e3-4215-9c29-572241d12f5e__1" type="other" other-type-definition="extended"/>
          <identity name="contrail:virtual-machine:default-domain__TestProject6D2DC8__snat_b243f11f-1c14-4237-b4b7-7fb86f6034dd_2be83366-32e3-4215-9c29-572241d12f5e__2" type="other" other-type-definition="extended"/>

On a working ifmap node.
root@b5s27:/tmp# grep 7fb86f6034dd_2be83366-32e3-4215-9c29-572241d12f5e /tmp/ifmap-output
root@b5s27:/tmp#

[OpenContrail Admin (ci-admin-f)]

bug update...

[Ashish Ranjan (aranjan-n)]

Need to repro this