Contrail is not sending remote mac delete notification for all the MAC on the logical switch when mac ages out / cleared in remote TOR

[Arun Paul]

This contrail cluster has 3 Control nodes , 2 TSNs and 2 Vrouter .

The full testbed.py can be got from Control Node ip 10.94.63.102

root@NTTC-Contrail-1:~# vi /opt/contrail/utils/fabfile/testbeds/testbed.py

This issue was brought to the notice with Contrail team where we saw that the remotely learnt mac was still seen in the TOR even though no local mac was present in Remote TOR .
The OVSDB database still had Remote MAC in the current TOR .

We went through the TSN introspect and made sure that Contrail is not holding any MAC .
A PR was raised against QFX
https://gnats.juniper.net/web/default/1253690#audit_tab

On debugging the PR we found that the contrail is not sending delete notification for all the deleted MAC in the Remote TOR
Please refer the recent Audit trail for all the outputs including the introspect output in the PR .

Now to make this simple ,I am just using 2 TOR's and I am sending traffic only on one logical switch and making the TORs learn the local and remote MAC .

TOR1

root@vdc-vcf-s1# run show ethernet-switching table vlan-name Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)

Ethernet switching table : 20 entries, 20 learned
Routing instance : default-switch
    Vlan MAC MAC Age Logical
    name address flags interface
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:01 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:02 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:03 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:04 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:05 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:06 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:07 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:08 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:09 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:0a D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:01 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:02 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:03 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:04 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:05 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:06 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:07 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:08 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:09 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:0a DO - vtep.32770

{master:1}[edit]
root@vdc-vcf-s1#

TOR2

root@vdc-vcf-s2# run show ethernet-switching table vlan-name Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)

Ethernet switching table : 20 entries, 20 learned
Routing instance : default-switch
    Vlan MAC MAC Age Logical
    name address flags interface
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:01 DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:02 DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:03 DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:04 DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:05 DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:06 DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:07 DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:08 DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:09 DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:0a DO - vtep.32769
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:01 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:02 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:03 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:04 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:05 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:06 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:07 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:08 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:09 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:0a D - ae2.88

{master:0}[edit]

The logical switch for Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 is c2fa8c57-c868-4da4-b127-59b4c367fdbe

I am capturing the transaction by

root@sw-ubuntu28:/home/ovsdb/openvswitch/openvswitch/openvswitch-2.3.3/ovsdb# ./ovsdb-client monitor tcp:10.94.47.65:6645 hardware_vtep ALL >mointor_TOR1

Now I stop the traffic and clear the ethernet switching table on TOR2

root@vdc-vcf-s2# run clear ethernet-switching table vlan-name Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2

{master:0}[edit]
root@vdc-vcf-s2# run clear ethernet-switching table vlan-name Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2

{master:0}[edit]
root@vdc-vcf-s2#

Now if I grep for the delete on the logical switch c2fa8c57-c868-4da4-b127-59b4c367fdbe .I only see delete notification for 4 MAC's

root@sw-ubuntu28:/home/ovsdb/openvswitch/openvswitch/openvswitch-2.3.3/ovsdb# cat mointor_TOR1 | grep -i c2fa8c57-c868-4da4-b127-59b4c367fdbe | grep delete
c60b5d10-7dc0-411a-9050-c9032a3ec8d9 delete "00:20:84:00:00:06" "" fb31ff73-c385-44c9-916e-1c8d2e611169 c2fa8c57-c868-4da4-b127-59b4c367fdbe caf590fe-70ce-4b75-b30a-a2f7da4cbfb0
4c9ef2c7-3333-4868-9769-d65769b78989 delete "00:20:84:00:00:02" "" fb31ff73-c385-44c9-916e-1c8d2e611169 c2fa8c57-c868-4da4-b127-59b4c367fdbe 85f4c0b8-14b4-4023-9193-6d92f0649f85
64110436-abb0-41bf-ac75-7edca4c1ac21 delete "00:20:84:00:00:0a" "" fb31ff73-c385-44c9-916e-1c8d2e611169 c2fa8c57-c868-4da4-b127-59b4c367fdbe 8c635cca-038d-4566-b637-dc70e3140436
46f7fcbe-bda8-4608-b96f-1b75193a3035 delete "00:20:84:00:00:07" "" fb31ff73-c385-44c9-916e-1c8d2e611169 c2fa8c57-c868-4da4-b127-59b4c367fdbe d527dae0-47df-4ec5-af42-d8bcee219139
root@sw-ubuntu28:/home/ovsdb/openvswitch/openvswitch/openvswitch-2.3.3/ovsdb#

And only those MAC's are getting deleted on TOR1

root@vdc-vcf-s1# run show ethernet-switching table vlan-name Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2

MAC flags (S - static MAC, D - dynamic MAC, L - locally learned, P - Persistent static
           SE - statistics enabled, NM - non configured MAC, R - remote PE MAC, O - ovsdb MAC)

Ethernet switching table : 16 entries, 16 learned
Routing instance : default-switch
    Vlan MAC MAC Age Logical
    name address flags interface
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:01 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:02 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:03 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:04 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:05 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:06 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:07 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:08 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:09 D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:10:84:00:00:0a D - ae2.88
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:01 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:03 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:04 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:05 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:08 DO - vtep.32770
    Contrail-cf06decc-36e9-4b9c-8bf0-7e72b34251c2 00:20:84:00:00:09 DO - vtep.32770

{master:1}[edit]
root@vdc-vcf-s1#

[Hari Prasad Killi (haripk)]

When there is active-backup tor-agent configuration, when local MAC is deleted, that doesnt get propagated immediately. The backup tor-agent exports a route with a lower preference and that still remains (for 5 minutes). This is done to avoid traffic loss upon OVSDB connection switchover. Please check that the entry is removed after 5 minutes.

[Arun Paul (ampul)]

We even waited for more than 5 minutes and even for a day .Still the switch was showing remotely learnt mac through vtep.Even though the mac was not seen locally in the remote switch .
The OVSDB database still had Remote MAC in the current TOR .

[Sandeep Sridhar (ssandeep)]

Hi Hari/Arun,

  I think this if fixed as a part of https://bugs.launchpad.net/juniperopenstack/+bug/1651618

I remember replicating this issue and sharing the setup with you/Manish. I guess fix was committed into 3.1.2.

Hope this helps.

Greetings,
Sandeep.

[vivekananda shenoy (vshenoy83)]

Hi Arun,

Can you please verify if this is fixed in the latest 3.1 (http://10.84.5.120/github-build/R3.1/73/ubuntu-14-04/mitaka/contrail-install-packages_3.1.3.0-73~mitaka_all.deb) verify and close this bug ?

Regards
Vivek