14.1X53D26/Contrail: after restart ovsdb-server it takes more than 15+ min re-sync config / recover the traffic

[Praveen Shetty]

Hi Rahul,

Sudarsan just debugged this issue, and confirmed that its contrail issue..
Whenever we do "restart ovsdb-server" contrail is deleting and adding remote mac, that's why we are seeing traffic drop on the remote node

I will update the PR with the logs, please assign the PR accordingly

Thanks,
Praveen

vi /var/db/ovsdatabase:

OVSDB JSON 90 837ead3587d0db68d6c1d4df1c2913100fdc3bf8
{"Ucast_Macs_Remote":{"a4b9ae1f-82e2-4302-b95f-061c91053214":null},"_date":1431112256432}
OVSDB JSON 90 fe579f67deb6675b0e1f254fc32fb2e850ff181c
{"Ucast_Macs_Remote":{"1af34058-aba3-4f6e-8bf4-fb94d0da4bbf":null},"_date":1431112256434}
OVSDB JSON 90 3c3224db3c2d9804647e69ce00c2e838cc4b50ba
{"Ucast_Macs_Remote":{"e585759d-4600-43d4-9ddc-8e4cf4841d30":null},"_date":1431112256435}
OVSDB JSON 90 fc86b598c88490a90454b0c07d0be966c953cc3c
{"Ucast_Macs_Remote":{"07fd2ba1-1318-4d53-a0e7-d4725cbf4393":null},"_date":1431112256440}
OVSDB JSON 90 449c913b68cfb882933223ef384c31eb9736c8c1
{"Ucast_Macs_Remote":{"db267c1d-797d-4b8c-971e-9cbad9efd387":null},"_date":1431112256453}
OVSDB JSON 236 9961aac995a4e1f3223dbdd229c18f1bc7da95f7
{"Ucast_Macs_Remote":{"98e48ac6-d179-4de1-ae1a-47b3dc9e3812":{"logical_switch":["uuid","fbc8197e-82c0-49c2-bb50-b0be0a03ba74"],"MAC":"00:08:23:10:02:2b","locator":["uuid","fb41
937a-ebd6-4c07-a15e-6b7f7451f435"]}},"_date":1431112348736}
OVSDB JSON 236 150d9b675bfc2c533073c9d1ca4fa1e9233a08f7
{"Ucast_Macs_Remote":{"b32d83a4-1882-430b-b5db-c69ca5e40896":{"logical_switch":["uuid","fbc8197e-82c0-49c2-bb50-b0be0a03ba74"],"MAC":"00:08:23:10:00:b4","locator":["uuid","fb41
937a-ebd6-4c07-a15e-6b7f7451f435"]}},"_date":1431112348740}
OVSDB JSON 236 453b1346eb0ad0a7e852cfe785c9864e5bea8a70
{"Ucast_Macs_Remote":{"318cf6f9-b904-4ff6-92af-a7b0ef7da4be":{"logical_switch":["uuid","fbc8197e-82c0-49c2-bb50-b0be0a03ba74"],"MAC":"00:08:23:10:03:25","locator":["uuid","fb41
937a-ebd6-4c07-a15e-6b7f7451f435"]}},"_date":1431112348741}
OVSDB JSON 236 b7ffaa212551b7b485aa1865e3cdc333f9038183
{"Ucast_Macs_Remote":{"3be71b7b-9aee-4f59-bbb4-bb258350a308":{"logical_switch":["uuid","fbc8197e-82c0-49c2-bb50-b0be0a03ba74"],"MAC":"00:08:23:10:01:31","locator":["uuid","fb41
937a-ebd6-4c07-a15e-6b7f7451f435"]}},"_date":1431112348744}
OVSDB JSON 236 fd73e08756673e6f5373833f9c0d13d9f9e865be
{"Ucast_Macs_Remote":{"619b4855-9d1f-48d8-9ce7-fab1fb9cf4da":{"logical_switch":["uuid","fbc8197e-82c0-49c2-bb50-b0be0a03ba74"],"MAC":"00:08:23:10:03:a2","locator":["uuid","fb41
937a-ebd6-4c07-a15e-6b7f7451f435"]}},"_date":1431112348744}
OVSDB JSON 236 bba2a93abd9d399b7f7c0c5d39225d21e6fbcb1d
{"Ucast_Macs_Remote":{"9ec64c75-5b95-43c6-bffb-6911239c95ac":{"logical_switch":["uuid","fbc8197e-82c0-49c2-bb50-b0be0a03ba74"],"MAC":"00:08:23:10:00:37","locator":["uuid","fb41

-------------------------------------------------------------------------------------------------------

after restart ovsdb-server it takes more than 15+ min re-sync config / recover the traffic

Traffic profile:
bi-dir L2 unicast traffic between DUT and 8 hw vteps at the rate of 1000pps

root@st-24q-p2-8> show ethernet-switching table summary
Total dynamic and static MAC addresses learned globally : 22776
Configured static MAC addresses learned globally : 14856

config:
4K LIFs
1K VNI
1k VTEP

have copied the gcore and ovsdb logs here

[pvshetty@glo-shell4 PR-1086240]$ pwd
/volume/dcg-systest/PRS/PR-1086240
[pvshetty@glo-shell4 PR-1086240]$ ls
ovs_core ovsdb.log.0.gz ovsdb.log.12.gz ovsdb.log.15.gz ovsdb.log.18.gz ovsdb.log.3.gz ovsdb.log.6.gz ovsdb.log.9.gz
ovs_core-1 ovsdb.log.10.gz ovsdb.log.13.gz ovsdb.log.16.gz ovsdb.log.1.gz ovsdb.log.4.gz ovsdb.log.7.gz
ovsdb.log ovsdb.log.11.gz ovsdb.log.14.gz ovsdb.log.17.gz ovsdb.log.2.gz ovsdb.log.5.gz ovsdb.log.8.gz

[Hari Prasad Killi (haripk)]

This was the behaviour in R2.1. An audit mechanism has been added as part of R2.2. Please recheck with an R2.2 build.

[Ashish Ranjan (aranjan-n)]

This needs to be retested on R2.20 build. Marking it fix committed for these scopes.