contrail-control crashed at XmppStateMachine::ProcessStreamHeaderMessage

[Sandip Dey]

Build:R3.2 build11 ubuntu-14.04

 Logs saved at http://10.204.216.50/Docs/bugs/<bug-id>

BT
===
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-control'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 size (this=0x78) at /usr/include/c++/4.8/bits/basic_string.h:716
716 { return _M_rep()->_M_length; }
(gdb) bt
#0 size (this=0x78) at /usr/include/c++/4.8/bits/basic_string.h:716
#1 XmppConnection::SetTo (this=0x0, to="nodei12") at controller/src/xmpp/xmpp_connection.cc:200
#2 0x0000000000bbdcbb in XmppStateMachine::ProcessStreamHeaderMessage (this=0x17aaaa0, session=0x17b9270, msg=msg@entry=0x7fcb8831d000)
    at controller/src/xmpp/xmpp_state_machine.cc:1371
#3 0x0000000000bbe3f2 in XmppStateMachine::ProcessMessage (this=this@entry=0x17aaaa0, session=<optimized out>, msg=0x7fcb8831d000)
    at controller/src/xmpp/xmpp_state_machine.cc:1457
#4 0x0000000000bbe8fb in XmppStateMachine::DequeueEvent (this=0x17aaaa0, event=...) at controller/src/xmpp/xmpp_state_machine.cc:1586
#5 0x0000000000b72bc7 in operator() (a0=..., this=0x7fcb93e5baa0) at /usr/include/boost/function/function_template.hpp:767
#6 QueueTaskRunner<boost::intrusive_ptr<boost::statechart::event_base const>, WorkQueue<boost::intrusive_ptr<boost::statechart::event_base const> > >::RunQueue (this=0x7fcb0c3f3d80) at controller/src/base/queue_task.h:92
#7 0x00000000006d614f in TaskImpl::execute (this=0x7fcb949f3b40) at controller/src/base/task.cc:262
#8 0x00007fcb9bfbbb3a in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fcb9bfb7816 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fcb9bfb6f4b in ?? () from /usr/lib/libtbb.so.2
#11 0x00007fcb9bfb30ff in ?? () from /usr/lib/libtbb.so.2
#12 0x00007fcb9bfb32f9 in ?? () from /usr/lib/libtbb.so.2
#13 0x00007fcb9c1d7184 in start_thread (arg=0x7fcb93e5c700) at pthread_create.c:312
#14 0x00007fcb9b2a837d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27273
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27274
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27273
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27274
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27273
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/27274
Submitter: Ananth Suryanarayana (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27273
Committed: http://github.org/Juniper/contrail-controller/commit/f4afa7e2f13e9f1dde6bec92584b934b696a4ca7
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit f4afa7e2f13e9f1dde6bec92584b934b696a4ca7
Author: Ananth Suryanarayana <email address hidden>
Date: Wed Dec 14 11:03:28 2016 -0800

Do not access connection from cleared session

We enqueue XMPP messages to a queue and process. However, this enqueue is done
off IO Reader task. But from main thread, tcp event can cause OnSessionEvent()
which could potentially enqueue TcpClose() to the state machine ahead of
the event that processes the message itself

This bug possibly got exposed due to the change in timings due to commit
8fef2aa03522465f311ff3bdf11bfcdc7078366e for bug fix 1643486

Hence drop processing xmpp message if session is not associated with the
connection. We alaready delete the session through an event in order to
serialize access to session pointer. Tcp also protects TcpSession using
intrusive pointer TcpSessionPtr.

Change-Id: I717a31e673bbdbeef9a36832ee9f9df77fbe54aa
Closes-Bug: 1649195

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/27274
Committed: http://github.org/Juniper/contrail-controller/commit/9a27b2d1a58ce0ff062290cc83e9552d7a9fefe2
Submitter: Zuul (<email address hidden>)
Branch: master

commit 9a27b2d1a58ce0ff062290cc83e9552d7a9fefe2
Author: Ananth Suryanarayana <email address hidden>
Date: Wed Dec 14 11:03:28 2016 -0800

Do not access connection from cleared session

We enqueue XMPP messages to a queue and process. However, this enqueue is done
off IO Reader task. But from main thread, tcp event can cause OnSessionEvent()
which could potentially enqueue TcpClose() to the state machine ahead of
the event that processes the message itself

This bug possibly got exposed due to the change in timings due to commit
8fef2aa03522465f311ff3bdf11bfcdc7078366e for bug fix 1643486

Hence drop processing xmpp message if session is not associated with the
connection. We alaready delete the session through an event in order to
serialize access to session pointer. Tcp also protects TcpSession using
intrusive pointer TcpSessionPtr.

Change-Id: I717a31e673bbdbeef9a36832ee9f9df77fbe54aa
Closes-Bug: 1649195

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/44098
Submitter: Ananth Suryanarayana (<email address hidden>)

[Himanshu (bhimanshu)]

@Ananth, is there an ETA when the next 3.1 build will be available which will include the fix for this bug?

[Himanshu (bhimanshu)]

@Anath, can you please answer the Q in At#17 ?

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/44098
Committed: http://github.com/Juniper/contrail-controller/commit/2e8c3778da38433585f83fb0b7cd39a74b2d9878
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 2e8c3778da38433585f83fb0b7cd39a74b2d9878
Author: Ananth Suryanarayana <email address hidden>
Date: Mon Jun 25 11:06:37 2018 -0700

Do not access connection from cleared session

We enqueue XMPP messages to a queue and process. However, this enqueue
is done off IO Reader task. But from main thread, tcp event can cause
OnSessionEvent() which could potentially enqueue TcpClose() to the state
machine ahead of the event that processes the message itself

This bug possibly got exposed due to the change in timings due to commit
8fef2aa03522465f311ff3bdf11bfcdc7078366e for bug fix 1643486

Hence drop processing xmpp message if session is not associated with the
connection. We alaready delete the session through an event in order to
serialize access to session pointer. Tcp also protects TcpSession using
intrusive pointer TcpSessionPtr.

Change-Id: Ib3b49f6102440e582def20bff134690f6e29462a
Closes-Bug: 1649195