SM:R3.2:Multi interface Setup: vrouter-agent core XmppClient::XmppClient(EventManager*, XmppChannelConfig const*)

[sundarkh]

SM:R3.2:Multi interface Setup: vrouter-agent core XmppClient::XmppClient(EventManager*, XmppChannelConfig const*)

1) Install SM R3.2 Build2 ; Reimage the target multi interface setup with ubuntu-14-04; Reimage succesful
2) Issue Provision ; Observe that compute ndoes core at vrouter-agent with XmppClient::XmppClient(EventManager*, XmppChannelConfig const*) as soon as the provision gets completed

root@nodea4:~# gdb /usr/bin/contrail-vrouter-agent /var/crashes/core.contrail-vroute.2926.nodea4.1479901978
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/contrail-vrouter-agent...(no debugging symbols found)...done.

warning: core file may not match specified executable file.
[New LWP 2951]
[New LWP 2945]
[New LWP 2947]
[New LWP 2926]
[New LWP 2944]
[New LWP 2949]
[New LWP 2950]
[New LWP 2948]
[New LWP 2946]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fa164459cc9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007fa164459cc9 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fa16445d0d8 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x00007fa164452b86 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#3 0x00007fa164452c32 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x00000000012cdf02 in TaskEntry::~TaskEntry() ()
#5 0x00000000012cdff1 in TaskGroup::~TaskGroup() ()
#6 0x00000000012ce4f7 in TaskScheduler::~TaskScheduler() ()
#7 0x00000000012d3ef1 in boost::scoped_ptr<TaskScheduler>::~scoped_ptr() ()
#8 0x00007fa16445f259 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#9 0x00007fa16445f2a5 in exit () from /lib/x86_64-linux-gnu/libc.so.6
#10 0x0000000000f8ca44 in XmppClient::XmppClient(EventManager*, XmppChannelConfig const*) ()
#11 0x0000000000e39238 in VNController::XmppServerConnect() ()
#12 0x0000000000e39ec8 in VNController::ApplyDiscoveryXmppServicesInternal(std::vector<DSResponse, std::allocator<DSResponse> >) ()
#13 0x0000000000e3b645 in VNController::ControllerWorkQueueProcess(boost::shared_ptr<ControllerWorkQueueData>) ()
#14 0x0000000000e3c03c in boost::detail::function::function_obj_invoker1<boost::_bi::bind_t<bool, boost::_mfi::mf1<bool, VNController, boost::shared_ptr<ControllerWorkQueueData> >, boost::_bi::list2<boost::_bi::value<VNController*>, boost::arg<1> > >, bool, boost::shared_ptr<ControllerWorkQueueData> >::invoke(boost::detail::function::function_buffer&, boost::shared_ptr<ControllerWorkQueueData>) ()
#15 0x0000000000e3edf4 in QueueTaskRunner<boost::shared_ptr<ControllerWorkQueueData>, WorkQueue<boost::shared_ptr<ControllerWorkQueueData> > >::RunQueue() ()
#16 0x00000000012d245f in TaskImpl::execute() ()
#17 0x00007fa165028b3a in ?? () from /usr/lib/libtbb.so.2
#18 0x00007fa165024816 in ?? () from /usr/lib/libtbb.so.2
#19 0x00007fa165023f4b in ?? () from /usr/lib/libtbb.so.2
#20 0x00007fa1650200ff in ?? () from /usr/lib/libtbb.so.2
#21 0x00007fa1650202f9 in ?? () from /usr/lib/libtbb.so.2
#22 0x00007fa165244182 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#23 0x00007fa16451d47d in clone () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) info reg
rax 0x0 0
rbx 0x7fa167076000 140331194998784
rcx 0xffffffffffffffff -1
rdx 0x6 6
rsi 0xb87 2951
rdi 0xb6e 2926
rbp 0x7fa1645a3830 0x7fa1645a3830
rsp 0x7fa14fbfcf18 0x7fa14fbfcf18
r8 0xfefefefefefefeff -72340172838076673
r9 0xfefefefefeff092d -72340172838074067
r10 0x8 8
r11 0x202 514
r12 0x12e9234 19829300
r13 0x143c000 21217280
r14 0x19109e0 26282464
r15 0x20cfa10 34404880
rip 0x7fa164459cc9 0x7fa164459cc9 <raise+57>
eflags 0x202 [ IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
(gdb)

vrouter-agent log in target
-----------------------------
Config file </etc/contrail/contrail-vrouter-agent.conf> parsing completed.
log4cplus:ERROR No appenders could be found for logger (SANDESH).
log4cplus:ERROR Please initialize the log4cplus system properly.
Config file </etc/contrail/contrail-vrouter-agent.conf> parsing completed.
log4cplus:ERROR No appenders could be found for logger (SANDESH).
log4cplus:ERROR Please initialize the log4cplus system properly.
Config file </etc/contrail/contrail-vrouter-agent.conf> parsing completed.
log4cplus:ERROR No appenders could be found for logger (SANDESH).
log4cplus:ERROR Please initialize the log4cplus system properly.
Config file </etc/contrail/contrail-vrouter-agent.conf> parsing completed.
log4cplus:ERROR No appenders could be found for logger (SANDESH).
log4cplus:ERROR Please initialize the log4cplus system properly.
Config file </etc/contrail/contrail-vrouter-agent.conf> parsing completed.

core is available in
--------------------
Host : 10.204.216.49
credentials : stack/stack@123
/home/stack/servermanagersanity/threedottwo/vrouteragentcore/crashes
[stack@anamika crashes]$ ls -lrt
total 452604
-rw------- 1 stack stack 115986432 Nov 23 18:03 core.contrail-vroute.2926.nodea4.1479901978
-rw------- 1 stack stack 115666944 Nov 23 18:03 core.contrail-vroute.4647.nodea4.1479901982
-rw------- 1 stack stack 115884032 Nov 23 18:03 core.contrail-vroute.3308.nodea4.1479901980
-rw------- 1 stack stack 115912704 Nov 23 18:03 core.contrail-vroute.4848.nodea4.1479901986
[stack@anamika crashes]$

WorkAround :
------------

Restart the vrouter agent process, it becomes active

[Abhay Joshi (abhayj)]

Why is this SM provisioning bug if vrouter-agent is crashing? Without changing any config file, if restarting vrouter-agent is working fine as suggested in workaround, we need to get someone in agent to look into agent crash. I don't have reason to think of this as SM provisioning issue. Please get someone in agent team to look into.

[Nipa (nipak)]

Please post contrail-vrouter-agent.conf and contrail-vrouter-agent.log

[Hari Prasad Killi (haripk)]

All the cores are due to the same issue, agent is doing an exit because of this :

"2016-12-01 Thu 15:02:42:324.781 PST nodei20 [Thread 140021160912640, Pid 3560]: Error : No such file or directory, while using cacert file : /etc/contrail/ssl/certs/ca-cert.pem”

root@nodei20:/etc/contrail/ssl/certs# ls -l
total 8
-rw-r--r-- 1 root root 1139 Dec 1 15:03 ca-cert.pem

It looks like agent is restarted while this file setup is in progress. Agent exits a few times before the file is available and later it is stable.

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26772
Submitter: Dheeraj Gautam (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26773
Submitter: Dheeraj Gautam (<email address hidden>)

[sundarkh (sundar-kh)]

Seen with mainline build 4.0.0.0-3015 also

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26772
Committed: http://github.org/Juniper/contrail-puppet/commit/515d3aa7cffc95e835a57f5a9c2d6c2006a4a2fd
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 515d3aa7cffc95e835a57f5a9c2d6c2006a4a2fd
Author: Dheeraj Gautam <email address hidden>
Date: Sat Dec 3 10:08:30 2016 -0800

vrouter-crash: put cert files before reboot

Closes-Bug: #1646498

configure cert files before reboot of compute. This ensures, vrouter finds
this files when it comes up.

Change-Id: Ib2449cbea3dc472106a373b4b3727b76f75580b4

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26773
Committed: http://github.org/Juniper/contrail-puppet/commit/90c5c9f776cd0bfcf25bb7478df7b25ef56025ad
Submitter: Zuul (<email address hidden>)
Branch: master

commit 90c5c9f776cd0bfcf25bb7478df7b25ef56025ad
Author: Dheeraj Gautam <email address hidden>
Date: Sat Dec 3 10:08:30 2016 -0800

vrouter-crash: put cert files before reboot

Closes-Bug: #1646498

configure cert files before reboot of compute. This ensures, vrouter finds
this files when it comes up.

Change-Id: Ib2449cbea3dc472106a373b4b3727b76f75580b4