Agent double free/mem corruption crash in XmppChannelMux::WriteReady

Bug #1531556 reported by amit surana
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
High
Hari Prasad Killi
R2.21.x
Fix Committed
High
Hari Prasad Killi
R2.22.x
Fix Committed
High
Hari Prasad Killi
R3.0
Fix Committed
High
Hari Prasad Killi
Trunk
Fix Committed
High
Hari Prasad Killi

Bug Description

contrail 2.21.1 #15

core file location: 10.84.5.112:/cs-shared/bugs/1531556/core*.1531556

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007feed69c6cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007feed69c6cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007feed69ca0d8 in __GI_abort () at abort.c:89
#2 0x00007feed6a03394 in __libc_message (do_abort=do_abort@entry=1, fmt=fmt@entry=0x7feed6b11b28 "*** Error in `%s': %s: 0x%s ***\n")
    at ../sysdeps/posix/libc_fatal.c:175
#3 0x00007feed6a0f66e in malloc_printerr (ptr=<optimized out>, str=0x7feed6b11c58 "double free or corruption (out)", action=1)
    at malloc.c:4996
#4 _int_free (av=<optimized out>, p=<optimized out>, have_lock=0) at malloc.c:3840
#5 0x0000000000d90a30 in operator() (a0=..., this=0x7fee92ffa980) at /usr/include/boost/function/function_template.hpp:767
#6 XmppChannelMux::WriteReady (this=0x7feeac5e9200, ec=...) at controller/src/xmpp/xmpp_channel_mux.cc:40
#7 0x0000000000d5b82d in XmppConnection::WriteReady (this=0x7feeac207c30) at controller/src/xmpp/xmpp_connection.cc:105
#8 0x0000000000d65ecc in XmppConnectionManager::DequeueSession (this=<optimized out>, tcp_session=...)
    at controller/src/xmpp/xmpp_connection_manager.cc:60
#9 0x0000000000d66e14 in operator() (a1=..., p=<optimized out>, this=<optimized out>) at /usr/include/boost/bind/mem_fn_template.hpp:165
#10 operator()<bool, boost::_mfi::mf1<bool, XmppConnectionManager, boost::intrusive_ptr<TcpSession> >, boost::_bi::list1<boost::intrusive_ptr<TcpSession>&> > (a=<synthetic pointer>, f=..., this=<optimized out>) at /usr/include/boost/bind/bind.hpp:303
#11 operator()<boost::intrusive_ptr<TcpSession> > (a1=..., this=<optimized out>) at /usr/include/boost/bind/bind_template.hpp:32
#12 boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, XmppConnectionManager, boost::intrusive_ptr<TcpSession> >, boost::_bi::list2<boost::_bi::value<XmppConnectionManager*>, boost::arg<1> > >, bool, boost::intrusive_ptr<TcpSession> >::invoke (function_obj_ptr=..., a0=...) at /usr/include/boost/function/function_template.hpp:132
#13 0x0000000000d67a4d in operator() (a0=..., this=0x7fee92ffaab0) at /usr/include/boost/function/function_template.hpp:767
#14 QueueTaskRunner<boost::intrusive_ptr<TcpSession>, WorkQueue<boost::intrusive_ptr<TcpSession> > >::RunQueue (this=0x21333f0)
    at controller/src/base/queue_task.h:81
#15 0x0000000000fdd8b0 in TaskImpl::execute (this=0x7feed0227d40) at controller/src/base/task.cc:238
#16 0x00007feed7595b3a in ?? () from /usr/lib/libtbb.so.2
#17 0x00007feed7591816 in ?? () from /usr/lib/libtbb.so.2
#18 0x00007feed7590f4b in ?? () from /usr/lib/libtbb.so.2
#19 0x00007feed758d0ff in ?? () from /usr/lib/libtbb.so.2
#20 0x00007feed758d2f9 in ?? () from /usr/lib/libtbb.so.2
#21 0x00007feed77b1182 in start_thread (arg=0x7fee92ffb700) at pthread_create.c:312
#22 0x00007feed6a8a47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Tags: vrouter soln
amit surana (asurana-t)
description: updated
amit surana (asurana-t)
information type: Proprietary → Public
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/17770
Submitter: Nipa Kumar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17770
Committed: http://github.org/Juniper/contrail-controller/commit/3c69334f0baed352680ca883e982537c7c4b3810
Submitter: Zuul
Branch: R3.0

commit 3c69334f0baed352680ca883e982537c7c4b3810
Author: Nipa Kumar <email address hidden>
Date: Wed Feb 24 14:27:35 2016 -0800

Enqueued WriteReadyCb results in accessing a deleted AgentDnsXmppChannel object.
We need to UnRegisterWriteReady when the object is deleted.

Change-Id: I1f1f39635f792327f91426d728ab70665bcaec94
Closes-Bug:1531556
Closes-Bug:1547735

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/17947
Submitter: Nipa Kumar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/17947
Committed: http://github.org/Juniper/contrail-controller/commit/3f31199f87df8a8b8af881cf6e1211f6c0f15586
Submitter: Zuul
Branch: master

commit 3f31199f87df8a8b8af881cf6e1211f6c0f15586
Author: Nipa Kumar <email address hidden>
Date: Wed Feb 24 14:27:35 2016 -0800

Enqueued WriteReadyCb results in accessing a deleted AgentDnsXmppChannel object.
We need to UnRegisterWriteReady when the object is deleted.

Change-Id: I1f1f39635f792327f91426d728ab70665bcaec94
Closes-Bug:1531556
Closes-Bug:1547735

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/18191
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/18192
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/18193
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/18191
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/18192
Submitter: Hari Prasad Killi (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/18191
Committed: http://github.org/Juniper/contrail-controller/commit/7324146b1c75ba98aee4a37be88c08e05f7386e0
Submitter: Zuul
Branch: R2.22.x

commit 7324146b1c75ba98aee4a37be88c08e05f7386e0
Author: Nipa Kumar <email address hidden>
Date: Wed Feb 24 14:27:35 2016 -0800

Enqueued WriteReadyCb results in accessing a deleted AgentDnsXmppChannel object.
We need to UnRegisterWriteReady when the object is deleted.

Marking xmpp_pubsub_test as flaky.

Change-Id: I1f1f39635f792327f91426d728ab70665bcaec94
Closes-Bug:1531556
Closes-Bug:1547735
(cherry picked from commit 3c69334f0baed352680ca883e982537c7c4b3810)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/18192
Committed: http://github.org/Juniper/contrail-controller/commit/b601c885c9675677065f2db52b151aee21970253
Submitter: Zuul
Branch: R2.20

commit b601c885c9675677065f2db52b151aee21970253
Author: Nipa Kumar <email address hidden>
Date: Wed Feb 24 14:27:35 2016 -0800

Enqueued WriteReadyCb results in accessing a deleted AgentDnsXmppChannel object.
We need to UnRegisterWriteReady when the object is deleted.

Marking xmpp_pubsub_test as flaky.

Change-Id: I1f1f39635f792327f91426d728ab70665bcaec94
Closes-Bug:1531556
Closes-Bug:1547735
(cherry picked from commit 3c69334f0baed352680ca883e982537c7c4b3810)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/18270
Submitter: Nipa Kumar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.20

Review in progress for https://review.opencontrail.org/18271
Submitter: Nipa Kumar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/18193
Submitter: Nipa Kumar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/18270
Committed: http://github.org/Juniper/contrail-controller/commit/eb16a497ce8af3a2382554c627d4e75c2bb5f54c
Submitter: Zuul
Branch: R2.22.x

commit eb16a497ce8af3a2382554c627d4e75c2bb5f54c
Author: Nipa Kumar <email address hidden>
Date: Tue Mar 8 15:02:26 2016 -0800

ReEnable xmpp_pubsub_test, cause of failure for an expired ssl cert. Set the expiry to 20 years from now.

Change-Id: I1126aeca600b51f3fb65def479bbc837012aa284
Closes-Bug:1531556
Closes-Bug:1547735

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/18271
Committed: http://github.org/Juniper/contrail-controller/commit/313ecc22b2091d9e2386e2cec5f6b2253e7217a1
Submitter: Zuul
Branch: R2.20

commit 313ecc22b2091d9e2386e2cec5f6b2253e7217a1
Author: Nipa Kumar <email address hidden>
Date: Tue Mar 8 15:02:26 2016 -0800

ReEnable xmpp_pubsub_test, cause of failure for an expired ssl cert. Set the expiry to 20 years from now.

Change-Id: I1126aeca600b51f3fb65def479bbc837012aa284
Closes-Bug:1531556
Closes-Bug:1547735

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/18193
Committed: http://github.org/Juniper/contrail-controller/commit/6177985408063961bd0cc777b8182a59b4adf40c
Submitter: Zuul
Branch: R2.21.x

commit 6177985408063961bd0cc777b8182a59b4adf40c
Author: Nipa Kumar <email address hidden>
Date: Wed Feb 24 14:27:35 2016 -0800

Enqueued WriteReadyCb results in accessing a deleted AgentDnsXmppChannel object.
We need to UnRegisterWriteReady when the object is deleted.

ReEnable xmpp_pubsub_test, cause of failure for an expired ssl cert. Set the expiry to 20 years from now.

Closes-Bug:1531556
Closes-Bug:1547735

Change-Id: I1f1f39635f792327f91426d728ab70665bcaec94

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/18285
Submitter: Nipa Kumar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/18286
Submitter: Nipa Kumar (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/18285
Committed: http://github.org/Juniper/contrail-controller/commit/86858f2cc0a9deb9b8a8c0a718e8c53f36517a6a
Submitter: Zuul
Branch: R3.0

commit 86858f2cc0a9deb9b8a8c0a718e8c53f36517a6a
Author: Nipa Kumar <email address hidden>
Date: Tue Mar 8 15:02:26 2016 -0800

ReEnable xmpp_pubsub_test, cause of failure for an expired ssl cert. Set the expiry to 20 years from now.

Change-Id: I1126aeca600b51f3fb65def479bbc837012aa284
Closes-Bug:1531556
Closes-Bug:1547735

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/18286
Committed: http://github.org/Juniper/contrail-controller/commit/9e9fa5a967074b558884f46d1bde44c35b6d746c
Submitter: Zuul
Branch: master

commit 9e9fa5a967074b558884f46d1bde44c35b6d746c
Author: Nipa Kumar <email address hidden>
Date: Tue Mar 8 15:02:26 2016 -0800

ReEnable xmpp_pubsub_test, cause of failure for an expired ssl cert. Set the expiry to 20 years from now.

Change-Id: I1126aeca600b51f3fb65def479bbc837012aa284
Closes-Bug:1531556
Closes-Bug:1547735

Changed in juniperopenstack:
milestone: r3.0-fcs → r3.1.0.0-fcs
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.