Control and DNS crashes @ __mpn_submul_1

[Sarath]

This seen multiple times on Sanity tests.
Both Control and DNS crashes with similar back-traces and so tracking both here.

root@ContrailVM:~#
root@ContrailVM:~# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent initializing (XMPP:control-node:10.87.36.10, XMPP:control-node:10.87.36.11, XMPP:dns-server:10.87.36.10, XMPP:dns-server:10.87.36.11 connection down)
contrail-vrouter-nodemgr active

root@ContrailVM:~#

root@5a10s31:/var/crashes# ls -ltrd *
-rw------- 1 contrail contrail 33841152 Apr 27 17:17 5a10s31_core.contrail-dns.24694.5a10s31.1493338647
-rw------- 1 contrail contrail 36265984 Apr 27 17:21 5a10s31_core.contrail-contro.24693.5a10s31.1493338900
drwxr-xr-x 2 root root 4096 Apr 27 19:30 saved
-rw------- 1 contrail contrail 40079360 Apr 28 07:38 core.contrail-dns.6117.5a10s31.1493390310
-rw------- 1 contrail contrail 38383616 Apr 28 07:39 core.contrail-dns.31841.5a10s31.1493390376
-rwxrwxrwx 1 root root 304749953 Apr 28 12:16 contrail-control
root@5a10s31:/var/crashes#

>>>>>>>> DNS

root@5a10s31:/var/crashes# gdb /usr/bin/contrail-dns core.contrail-dns.31841.5a10s31.1493390376
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/contrail-dns...done.
[New LWP 31891]
[New LWP 31841]
[New LWP 31889]
[New LWP 31893]
[New LWP 31892]
[New LWP 31890]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-dns'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007ff8fd8fec37 in __mpn_submul_1 () at ../sysdeps/x86_64/addmul_1.S:90
90 ../sysdeps/x86_64/addmul_1.S: No such file or directory.
(gdb) bt
#0 0x00007ff8fd8fec37 in __mpn_submul_1 () at ../sysdeps/x86_64/addmul_1.S:90
#1 0x00007ff8fd902028 in drand48 () at drand48.c:27
#2 0x0000000000000000 in ?? ()
(gdb)

>>>>> Control

root@5a10s31:/var/crashes# gdb /usr/bin/contrail-dns 5a10s31_core.contrail-dns.24694.5a10s31.1493338647
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/contrail-dns...done.
Illegal process-id: 5a10s31_core.contrail-dns.24694.5a10s31.1493338647.
[New LWP 24819]
[New LWP 24820]
[New LWP 24824]
[New LWP 24694]
[New LWP 24818]
[New LWP 24821]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-dns'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f67d2e3ac37 in __mpn_submul_1 () at ../sysdeps/x86_64/addmul_1.S:90
90 ../sysdeps/x86_64/addmul_1.S: No such file or directory.
(gdb) bt
#0 0x00007f67d2e3ac37 in __mpn_submul_1 () at ../sysdeps/x86_64/addmul_1.S:90
#1 0x00007f67d2e3e028 in drand48 () at drand48.c:27
#2 0x0000000000000000 in ?? ()
(gdb) quit
root@5a10s31:/var/crashes#

[Sarath (nsarath)]

nsarath@ubuntu-build04:/auto/cores/1687096$ ls -ltrd *
-rwxrwxrwx 1 nsarath test 67092480 Apr 28 12:40 Esxi-1-log.tar
-rwxrwxrwx 1 nsarath test 59494400 Apr 28 12:40 Kvm-1-log.tar
-rwxrwxrwx 1 nsarath test 67010560 Apr 28 12:40 Esxi-2-log.tar
-rwxrwxrwx 1 nsarath test 616273920 Apr 28 12:40 Ctrl-C-log.tar
-rwxrwxrwx 1 nsarath test 849694720 Apr 28 12:41 Ctrl-B-log.tar
-rwxrwxrwx 1 nsarath test 1142558720 Apr 28 12:41 Ctrl-A-log.tar
-rwxrwxrwx 1 nsarath test 523438080 Apr 28 12:42 Crashes.tar

[Jeba Paulaiyan (jebap)]

can u please send the .conf details along with any logs for both control-node and dns.

[Nischal Sheth (nsheth)]

Triggered by provisioning error which resulted in 2
IPs for the same vRouter name.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/30916
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/30919
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/30922
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/30916
Committed: http://github.com/Juniper/contrail-controller/commit/77fa8e6c2db2625bb71ad5e2d9116bb6df7c295f
Submitter: Zuul (<email address hidden>)
Branch: master

commit 77fa8e6c2db2625bb71ad5e2d9116bb6df7c295f
Author: Ananth Suryanarayana <email address hidden>
Date: Mon May 1 15:52:14 2017 -0700

Drop new xmpp connection for existing connection with same name

If the IP address is different, then it implies mis-configuration of
the xmpp agents. In such cases, drop new connection request and
retain existing connection intact

Since xmpp tasks run concurrently (across different IP addresses),
use a XmppConnectionManager scoped mutex to solve concurrency issues
among such parallel xmpp connection requests processing

Also increase severity of a couple of related log messages

Re-enable and fix few UTs that test this very exact scenario

Closes-Bug: #1687096

Change-Id: Ic7252060abfc4b34fa184ff163afc217a88bb40c

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/30919
Committed: http://github.com/Juniper/contrail-controller/commit/0d5d2cc122ee8da541599b7bfe884d8d89e247a1
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 0d5d2cc122ee8da541599b7bfe884d8d89e247a1
Author: Ananth Suryanarayana <email address hidden>
Date: Mon May 1 15:52:14 2017 -0700

Drop new xmpp connection for existing connection with same name

If the IP address is different, then it implies mis-configuration of
the xmpp agents. In such cases, drop new connection request and
retain existing connection intact

Since xmpp tasks run concurrently (across different IP addresses),
use a XmppConnectionManager scoped mutex to solve concurrency issues
among such parallel xmpp connection requests processing

Also increase severity of a couple of related log messages

Re-enable and fix few UTs that test this very exact scenario

Closes-Bug: #1687096

Change-Id: Ic7252060abfc4b34fa184ff163afc217a88bb40c

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/30922
Committed: http://github.com/Juniper/contrail-controller/commit/2a5ba530be83d8ec252f0c3f36bff5a038c80823
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 2a5ba530be83d8ec252f0c3f36bff5a038c80823
Author: Ananth Suryanarayana <email address hidden>
Date: Mon May 1 15:52:14 2017 -0700

Drop new xmpp connection for existing connection with same name

If the IP address is different, then it implies mis-configuration of
the xmpp agents. In such cases, drop new connection request and
retain existing connection intact

Since xmpp tasks run concurrently (across different IP addresses),
use a XmppConnectionManager scoped mutex to solve concurrency issues
among such parallel xmpp connection requests processing

Also increase severity of a couple of related log messages

Re-enable and fix few UTs that test this very exact scenario

Change-Id: Id65ea8ba9b4f94bf8a566408c6fcb2e66ee570d8
Closes-Bug: #1687096

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/32565
Submitter: Vinay Vithal Mahuli (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/32565
Committed: http://github.com/Juniper/contrail-controller/commit/eec7e2f43e6be6913e9b8a376363263989b8851f
Submitter: Zuul (<email address hidden>)
Branch: R3.2.3.x

commit eec7e2f43e6be6913e9b8a376363263989b8851f
Author: Ananth Suryanarayana <email address hidden>
Date: Mon May 1 15:52:14 2017 -0700

Drop new xmpp connection for existing connection with same name

If the IP address is different, then it implies mis-configuration of
the xmpp agents. In such cases, drop new connection request and
retain existing connection intact

Since xmpp tasks run concurrently (across different IP addresses),
use a XmppConnectionManager scoped mutex to solve concurrency issues
among such parallel xmpp connection requests processing

Also increase severity of a couple of related log messages

Re-enable and fix few UTs that test this very exact scenario

Closes-Bug: #1687096

Change-Id: Ic7252060abfc4b34fa184ff163afc217a88bb40c