| 2014-11-07 20:20:12 |
shajuvk |
description |
Lets create a bug w/ both cores and bt.
________________________________________
From: Shaju V.K
Sent: Friday, November 7, 2014 11:53 AM
To: Atul Moghe; Ashish Ranjan; Vedamurthy Ananth Joshi; Contrail Systems Virtual Router Team
Cc: Abhay Joshi; cf-test
Subject: RE: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
When I restarted the agent service it came up but during the sanity test : test_vm_with_sec_group, again failed.
Thanks,
Shaju
From: Atul Moghe
Sent: Friday, November 07, 2014 11:27 AM
To: Ashish Ranjan; Vedamurthy Ananth Joshi; Shaju V.K; Contrail Systems Virtual Router Team
Cc: Abhay Joshi; cf-test
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fb3de7e3fc8 in __GI_abort () at abort.c:89
#2 0x00007fb3de7d9a76 in __assert_fail_base (
fmt=0x7fb3de92b370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0xe8c7d5 "0", file=file@entry=0xf44d9a "controller/src/base/task.cc",
line=line@entry=238, function=function@entry=0xf46000 "virtual tbb::task* TaskImpl::execute()")
at assert.c:92
#3 0x00007fb3de7d9b22 in __GI___assert_fail (assertion=0xe8c7d5 "0",
file=0xf44d9a "controller/src/base/task.cc", line=238,
function=0xf46000 "virtual tbb::task* TaskImpl::execute()") at assert.c:101
#4 0x0000000000e5f477 in TaskImpl::execute() ()
#5 0x00007fb3df3b0b3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007fb3df3ac816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007fb3df3abf4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007fb3df3a80ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fb3df3a82f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fb3df5cc182 in start_thread (arg=0x7fb3d6c2b700) at pthread_create.c:312
#11 0x00007fb3de8a4fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
From: Ashish Ranjan <aranjan@juniper.net>
Date: Friday, November 7, 2014 11:24 AM
To: Vedamurthy Ananth Joshi <vjoshi@juniper.net>, Atul Moghe <amoghe@juniper.net>, "Shaju V.K" <shajuvk@juniper.net>, Contrail Systems Virtual Router Team <dl-contrail-vnsw@juniper.net>
Cc: Abhay Joshi <abhayj@juniper.net>, cf-test <cf-test@juniper.net>
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
Atul could you post the bt.
________________________________________
From: Vedamurthy Ananth Joshi
Sent: Friday, November 7, 2014 11:08 AM
To: Atul Moghe; Shaju V.K; Contrail Systems Virtual Router Team
Cc: Ashish Ranjan; Abhay Joshi; cf-test
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
+ vrouter team..
From: Atul Moghe <amoghe@juniper.net>
Date: Saturday, November 8, 2014 at 12:27 AM
To: "Shaju V.K" <shajuvk@juniper.net>
Cc: Ashish Ranjan <aranjan@juniper.net>, Abhay Joshi <abhayj@juniper.net>, cf-test <cf-test@juniper.net>
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
I see same crash in my setup working with latest nova for 1404. Need to work with agent team..
When I ran quick_sanity, first 8 test cases passed and agent crashed in test_svc_monitor_datqpath test case,
Can you confirm if agent crash is only with service chaining or with other test cases as well?
Thanks,
Atul
From: "Shaju V.K" <shajuvk@juniper.net>
Date: Friday, November 7, 2014 9:17 AM
To: Atul Moghe <amoghe@juniper.net>
Cc: Ashish Ranjan <aranjan@juniper.net>, Abhay Joshi <abhayj@juniper.net>, cf-test <cf-test@juniper.net>
Subject: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
Hi Atul,
One of the Vrouter-agent is down on multi node ubuntu-14.04 setup. Could you please check the node a5s9. There is an agent crash.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fb3de7e3fc8 in __GI_abort () at abort.c:89
#2 0x00007fb3de7d9a76 in __assert_fail_base (fmt=0x7fb3de92b370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xe8c7d5 "0",
file=file@entry=0xf44d9a "controller/src/base/task.cc", line=line@entry=238, function=function@entry=0xf46000 "virtual tbb::task* TaskImpl::execute()")
at assert.c:92
#3 0x00007fb3de7d9b22 in __GI___assert_fail (assertion=0xe8c7d5 "0", file=0xf44d9a "controller/src/base/task.cc", line=238,
function=0xf46000 "virtual tbb::task* TaskImpl::execute()") at assert.c:101
#4 0x0000000000e5f477 in TaskImpl::execute() ()
#5 0x00007fb3df3b0b3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007fb3df3ac816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007fb3df3abf4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007fb3df3a80ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fb3df3a82f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fb3df5cc182 in start_thread (arg=0x7fb3d6c2b700) at pthread_create.c:312
#11 0x00007fb3de8a4fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)
root@a5s9:/var/log/contrail# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent failed
contrail-vrouter-nodemgr active
========Run time service failures=============
/var/crashes/core.contrail-vroute.1616.a5s9.1415379503
/var/crashes/core.contrail-vroute.6419.a5s9.1415379518
/var/crashes/core.contrail-vroute.6912.a5s9.1415379526
/var/crashes/core.contrail-vroute.5464.a5s9.1415379506
/var/crashes/core.contrail-vroute.5957.a5s9.1415379512
Testbed:
=========
host1 = 'root@10.84.14.6'
host2 = 'root@10.84.14.7'
host3 = 'root@10.84.14.8'
host4 = 'root@10.84.14.9'
host5 = 'root@10.84.14.10'
#Role definition of the hosts.
env.roledefs = {
'all': [host1, host2, host3, host4, host5],
'cfgm': [host1, host2],
'openstack': [host3],
'control': [host1, host2],
'compute': [host4, host5],
'collector': [host1, host2, host3],
'webui': [host1],
'database': [host1, host2, host3],
'build': [host_build],
}
env.hostnames = {
'all': ['a5s6', 'a5s7', 'a5s8','a5s9', 'a5s10']
Thanks,
Shaju |
When I restarted the agent service it came up but during the sanity test : test_vm_with_sec_group, again failed.
Thanks,
Shaju
From: Atul Moghe
Sent: Friday, November 07, 2014 11:27 AM
To: Ashish Ranjan; Vedamurthy Ananth Joshi; Shaju V.K; Contrail Systems Virtual Router Team
Cc: Abhay Joshi; cf-test
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fb3de7e3fc8 in __GI_abort () at abort.c:89
#2 0x00007fb3de7d9a76 in __assert_fail_base (
fmt=0x7fb3de92b370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=assertion@entry=0xe8c7d5 "0", file=file@entry=0xf44d9a "controller/src/base/task.cc",
line=line@entry=238, function=function@entry=0xf46000 "virtual tbb::task* TaskImpl::execute()")
at assert.c:92
#3 0x00007fb3de7d9b22 in __GI___assert_fail (assertion=0xe8c7d5 "0",
file=0xf44d9a "controller/src/base/task.cc", line=238,
function=0xf46000 "virtual tbb::task* TaskImpl::execute()") at assert.c:101
#4 0x0000000000e5f477 in TaskImpl::execute() ()
#5 0x00007fb3df3b0b3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007fb3df3ac816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007fb3df3abf4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007fb3df3a80ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fb3df3a82f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fb3df5cc182 in start_thread (arg=0x7fb3d6c2b700) at pthread_create.c:312
#11 0x00007fb3de8a4fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
From: Ashish Ranjan <aranjan@juniper.net>
Date: Friday, November 7, 2014 11:24 AM
To: Vedamurthy Ananth Joshi <vjoshi@juniper.net>, Atul Moghe <amoghe@juniper.net>, "Shaju V.K" <shajuvk@juniper.net>, Contrail Systems Virtual Router Team <dl-contrail-vnsw@juniper.net>
Cc: Abhay Joshi <abhayj@juniper.net>, cf-test <cf-test@juniper.net>
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
Atul could you post the bt.
________________________________________
From: Vedamurthy Ananth Joshi
Sent: Friday, November 7, 2014 11:08 AM
To: Atul Moghe; Shaju V.K; Contrail Systems Virtual Router Team
Cc: Ashish Ranjan; Abhay Joshi; cf-test
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
+ vrouter team..
From: Atul Moghe <amoghe@juniper.net>
Date: Saturday, November 8, 2014 at 12:27 AM
To: "Shaju V.K" <shajuvk@juniper.net>
Cc: Ashish Ranjan <aranjan@juniper.net>, Abhay Joshi <abhayj@juniper.net>, cf-test <cf-test@juniper.net>
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
I see same crash in my setup working with latest nova for 1404. Need to work with agent team..
When I ran quick_sanity, first 8 test cases passed and agent crashed in test_svc_monitor_datqpath test case,
Can you confirm if agent crash is only with service chaining or with other test cases as well?
Thanks,
Atul
From: "Shaju V.K" <shajuvk@juniper.net>
Date: Friday, November 7, 2014 9:17 AM
To: Atul Moghe <amoghe@juniper.net>
Cc: Ashish Ranjan <aranjan@juniper.net>, Abhay Joshi <abhayj@juniper.net>, cf-test <cf-test@juniper.net>
Subject: Ubuntu-14-04 -2443-vrouter-agent down in multi-node
Hi Atul,
One of the Vrouter-agent is down on multi node ubuntu-14.04 setup. Could you please check the node a5s9. There is an agent crash.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fb3de7e3fc8 in __GI_abort () at abort.c:89
#2 0x00007fb3de7d9a76 in __assert_fail_base (fmt=0x7fb3de92b370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xe8c7d5 "0",
file=file@entry=0xf44d9a "controller/src/base/task.cc", line=line@entry=238, function=function@entry=0xf46000 "virtual tbb::task* TaskImpl::execute()")
at assert.c:92
#3 0x00007fb3de7d9b22 in __GI___assert_fail (assertion=0xe8c7d5 "0", file=0xf44d9a "controller/src/base/task.cc", line=238,
function=0xf46000 "virtual tbb::task* TaskImpl::execute()") at assert.c:101
#4 0x0000000000e5f477 in TaskImpl::execute() ()
#5 0x00007fb3df3b0b3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007fb3df3ac816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007fb3df3abf4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007fb3df3a80ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fb3df3a82f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fb3df5cc182 in start_thread (arg=0x7fb3d6c2b700) at pthread_create.c:312
#11 0x00007fb3de8a4fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)
root@a5s9:/var/log/contrail# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent failed
contrail-vrouter-nodemgr active
========Run time service failures=============
/var/crashes/core.contrail-vroute.1616.a5s9.1415379503
/var/crashes/core.contrail-vroute.6419.a5s9.1415379518
/var/crashes/core.contrail-vroute.6912.a5s9.1415379526
/var/crashes/core.contrail-vroute.5464.a5s9.1415379506
/var/crashes/core.contrail-vroute.5957.a5s9.1415379512
Testbed:
=========
host1 = 'root@10.84.14.6'
host2 = 'root@10.84.14.7'
host3 = 'root@10.84.14.8'
host4 = 'root@10.84.14.9'
host5 = 'root@10.84.14.10'
#Role definition of the hosts.
env.roledefs = {
'all': [host1, host2, host3, host4, host5],
'cfgm': [host1, host2],
'openstack': [host3],
'control': [host1, host2],
'compute': [host4, host5],
'collector': [host1, host2, host3],
'webui': [host1],
'database': [host1, host2, host3],
'build': [host_build],
}
env.hostnames = {
'all': ['a5s6', 'a5s7', 'a5s8','a5s9', 'a5s10']
Thanks,
Shaju |
|
