[3.0 2717]: TOR Scale: Tor Agent crash @ OVSDB::LogicalSwitchTable::OvsdbMcastRemoteMacNotify

[chhandak]

Observed this crash with scale setup while deleting tor scale config

Backtrace
------------------
(gdb) bt
#0 strlen () at ../sysdeps/x86_64/strlen.S:106
#1 0x00007f2e1c851d35 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x0000000000a78cb5 in OVSDB::LogicalSwitchTable::OvsdbMcastRemoteMacNotify(OVSDB::OvsdbClientIdl::Op, ovsdb_idl_row*) ()
#3 0x0000000000a8eb89 in ?? ()
#4 0x0000000000a8ef1a in ?? ()
#5 0x0000000000a8dffe in ?? ()
#6 0x0000000000a8deb5 in ?? ()
#7 0x0000000000a8db3c in ?? ()
#8 0x0000000000a8d383 in ovsdb_idl_msg_process ()
#9 0x0000000000a0a2d8 in OVSDB::OvsdbClientIdl::ProcessMessage(OVSDB::OvsdbClientIdl::OvsdbMsg*) ()
#10 0x0000000000a0ed0f in QueueTaskRunner<OVSDB::OvsdbClientIdl::OvsdbMsg*, WorkQueue<OVSDB::OvsdbClientIdl::OvsdbMsg*> >::Run() ()
#11 0x0000000000f5686c in TaskImpl::execute() ()
#12 0x00007f2e1cabab3a in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f2e1cab6816 in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f2e1cab5f4b in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f2e1cab20ff in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f2e1cab22f9 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f2e1ccd6182 in start_thread (arg=0x7f2e14d70700) at pthread_create.c:312
#18 0x00007f2e1bfaf47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[chhandak (chhandak)]

Logs saved at http://mayamruga.englab.juniper.net/bugs/1550499

[chhandak (chhandak)]

Observed in 2720 Also

(gdb) bt
#0 strlen () at ../sysdeps/x86_64/strlen.S:106
#1 0x00007f3cbe927d35 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x0000000000a7bd15 in OVSDB::LogicalSwitchTable::OvsdbMcastRemoteMacNotify(OVSDB::OvsdbClientIdl::Op, ovsdb_idl_row*) ()
#3 0x0000000000a91aa9 in ?? ()
#4 0x0000000000a91e3a in ?? ()
#5 0x0000000000a90f1e in ?? ()
#6 0x0000000000a90dd5 in ?? ()
#7 0x0000000000a90a5c in ?? ()
#8 0x0000000000a902a3 in ovsdb_idl_msg_process ()
#9 0x0000000000a0d3c8 in OVSDB::OvsdbClientIdl::ProcessMessage(OVSDB::OvsdbClientIdl::OvsdbMsg*) ()
#10 0x0000000000a11dff in QueueTaskRunner<OVSDB::OvsdbClientIdl::OvsdbMsg*, WorkQueue<OVSDB::OvsdbClientIdl::OvsdbMsg*> >::Run() ()
#11 0x0000000000f5993c in TaskImpl::execute() ()
#12 0x00007f3cbeb90b3a in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f3cbeb8c816 in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f3cbeb8bf4b in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f3cbeb880ff in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f3cbeb882f9 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f3cbedac182 in start_thread (arg=0x7f3cb4e40700) at pthread_create.c:312
#18 0x00007f3cbe08547d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)
0 nodei6 1* nodei7 2 nodei8 3

[chhandak (chhandak)]

Observed in 2722 also

Backtrace
---------------
#0 strlen () at ../sysdeps/x86_64/strlen.S:106
#1 0x00007f0a7511cd35 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x0000000000a7bd65 in OVSDB::LogicalSwitchTable::OvsdbMcastRemoteMacNotify(OVSDB::OvsdbClientIdl::Op, ovsdb_idl_row*) ()
#3 0x0000000000a91af9 in ?? ()
#4 0x0000000000a91e8a in ?? ()
#5 0x0000000000a90f6e in ?? ()
#6 0x0000000000a90e25 in ?? ()
#7 0x0000000000a90aac in ?? ()
#8 0x0000000000a902f3 in ovsdb_idl_msg_process ()
#9 0x0000000000a0d418 in OVSDB::OvsdbClientIdl::ProcessMessage(OVSDB::OvsdbClientIdl::OvsdbMsg*) ()
#10 0x0000000000a11e4f in QueueTaskRunner<OVSDB::OvsdbClientIdl::OvsdbMsg*, WorkQueue<OVSDB::OvsdbClientIdl::OvsdbMsg*> >::Run() ()
#11 0x0000000000f59a7c in TaskImpl::execute() ()
#12 0x00007f0a75385b3a in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f0a75381816 in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f0a75380f4b in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f0a7537d0ff in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f0a7537d2f9 in ?? () from /usr/lib/libtbb.so.2
#17 0x00007f0a755a1182 in start_thread (arg=0x7f0a6c637700) at pthread_create.c:312
#18 0x00007f0a7487a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[Prabhjot Singh Sethi (prabhjot)]

issue happens on physical locator set delete where freed memory address is not reset to NULL causing free memory read and crash

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18666
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18667
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18669
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18670
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18671
Submitter: Prabhjot Singh Sethi (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18666
Committed: http://github.org/Juniper/contrail-third-party/commit/75011e1d098c932c5c369729c7dee48ebab837c3
Submitter: Zuul
Branch: master

commit 75011e1d098c932c5c369729c7dee48ebab837c3
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Mar 23 15:41:20 2016 +0530

Fix freed memory access in ovsdb client

Issue:
------
unparse while deleting row free's memory but doesnot
reset the pointer holding that memory, causing free'd
memory read in ToR-Agent and causing crash if the same
memory is reused before read for some other object

Fix:
----
reset pointer holding memory once the memory is free'd

Closes-Bug: 1550492
Change-Id: I11c65c5fac72ca8fbbb32610709827e906cd451a

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18667
Committed: http://github.org/Juniper/contrail-third-party/commit/34c3a4be7831dc702baac8b3bcb5fc2b7046f3e5
Submitter: Zuul
Branch: R3.0

commit 34c3a4be7831dc702baac8b3bcb5fc2b7046f3e5
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Mar 23 15:41:20 2016 +0530

Fix freed memory access in ovsdb client

Issue:
------
unparse while deleting row free's memory but doesnot
reset the pointer holding that memory, causing free'd
memory read in ToR-Agent and causing crash if the same
memory is reused before read for some other object

Fix:
----
reset pointer holding memory once the memory is free'd

Closes-Bug: 1550492
Change-Id: I11c65c5fac72ca8fbbb32610709827e906cd451a
(cherry picked from commit 75011e1d098c932c5c369729c7dee48ebab837c3)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18669
Committed: http://github.org/Juniper/contrail-third-party/commit/dfc2d92913aa7b805220907fc304d2b7d0494a88
Submitter: Zuul
Branch: R2.20

commit dfc2d92913aa7b805220907fc304d2b7d0494a88
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Mar 23 15:41:20 2016 +0530

Fix freed memory access in ovsdb client

Issue:
------
unparse while deleting row free's memory but doesnot
reset the pointer holding that memory, causing free'd
memory read in ToR-Agent and causing crash if the same
memory is reused before read for some other object

Fix:
----
reset pointer holding memory once the memory is free'd

Closes-Bug: 1550492
Change-Id: I11c65c5fac72ca8fbbb32610709827e906cd451a
(cherry picked from commit 75011e1d098c932c5c369729c7dee48ebab837c3)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18671
Committed: http://github.org/Juniper/contrail-third-party/commit/324be342434a224ac667ccac2613d9c120cde17c
Submitter: Zuul
Branch: R2.22.x

commit 324be342434a224ac667ccac2613d9c120cde17c
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Mar 23 15:41:20 2016 +0530

Fix freed memory access in ovsdb client

Issue:
------
unparse while deleting row free's memory but doesnot
reset the pointer holding that memory, causing free'd
memory read in ToR-Agent and causing crash if the same
memory is reused before read for some other object

Fix:
----
reset pointer holding memory once the memory is free'd

Closes-Bug: 1550492
Change-Id: I11c65c5fac72ca8fbbb32610709827e906cd451a
(cherry picked from commit 75011e1d098c932c5c369729c7dee48ebab837c3)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18670
Committed: http://github.org/Juniper/contrail-third-party/commit/bd8bf583a2fd85527238800768ab308d3fbd46d7
Submitter: Zuul
Branch: R2.21.x

commit bd8bf583a2fd85527238800768ab308d3fbd46d7
Author: Prabhjot Singh Sethi <email address hidden>
Date: Wed Mar 23 15:41:20 2016 +0530

Fix freed memory access in ovsdb client

Issue:
------
unparse while deleting row free's memory but doesnot
reset the pointer holding that memory, causing free'd
memory read in ToR-Agent and causing crash if the same
memory is reused before read for some other object

Fix:
----
reset pointer holding memory once the memory is free'd

Closes-Bug: 1550492
Change-Id: I11c65c5fac72ca8fbbb32610709827e906cd451a
(cherry picked from commit 75011e1d098c932c5c369729c7dee48ebab837c3)