Ubuntu-14-04-agent crash:/nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.

[shajuvk]

When I restarted the agent service it came up but during the sanity test : test_vm_with_sec_group, again failed.

Thanks,
Shaju

From: Atul Moghe
Sent: Friday, November 07, 2014 11:27 AM
To: Ashish Ranjan; Vedamurthy Ananth Joshi; Shaju V.K; Contrail Systems Virtual Router Team
Cc: Abhay Joshi; cf-test
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fb3de7e3fc8 in __GI_abort () at abort.c:89
#2 0x00007fb3de7d9a76 in __assert_fail_base (
    fmt=0x7fb3de92b370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0xe8c7d5 "0", file=file@entry=0xf44d9a "controller/src/base/task.cc",
    line=line@entry=238, function=function@entry=0xf46000 "virtual tbb::task* TaskImpl::execute()")
    at assert.c:92
#3 0x00007fb3de7d9b22 in __GI___assert_fail (assertion=0xe8c7d5 "0",
    file=0xf44d9a "controller/src/base/task.cc", line=238,
    function=0xf46000 "virtual tbb::task* TaskImpl::execute()") at assert.c:101
#4 0x0000000000e5f477 in TaskImpl::execute() ()
#5 0x00007fb3df3b0b3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007fb3df3ac816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007fb3df3abf4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007fb3df3a80ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fb3df3a82f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fb3df5cc182 in start_thread (arg=0x7fb3d6c2b700) at pthread_create.c:312
#11 0x00007fb3de8a4fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

From: Ashish Ranjan <email address hidden>
Date: Friday, November 7, 2014 11:24 AM
To: Vedamurthy Ananth Joshi <email address hidden>, Atul Moghe <email address hidden>, "Shaju V.K" <email address hidden>, Contrail Systems Virtual Router Team <email address hidden>
Cc: Abhay Joshi <email address hidden>, cf-test <email address hidden>
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node

Atul could you post the bt.

________________________________________
From: Vedamurthy Ananth Joshi
Sent: Friday, November 7, 2014 11:08 AM
To: Atul Moghe; Shaju V.K; Contrail Systems Virtual Router Team
Cc: Ashish Ranjan; Abhay Joshi; cf-test
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node

+ vrouter team..

From: Atul Moghe <email address hidden>
Date: Saturday, November 8, 2014 at 12:27 AM
To: "Shaju V.K" <email address hidden>
Cc: Ashish Ranjan <email address hidden>, Abhay Joshi <email address hidden>, cf-test <email address hidden>
Subject: Re: Ubuntu-14-04 -2443-vrouter-agent down in multi-node

I see same crash in my setup working with latest nova for 1404. Need to work with agent team..
When I ran quick_sanity, first 8 test cases passed and agent crashed in test_svc_monitor_datqpath test case,
Can you confirm if agent crash is only with service chaining or with other test cases as well?

Thanks,
Atul

From: "Shaju V.K" <email address hidden>
Date: Friday, November 7, 2014 9:17 AM
To: Atul Moghe <email address hidden>
Cc: Ashish Ranjan <email address hidden>, Abhay Joshi <email address hidden>, cf-test <email address hidden>
Subject: Ubuntu-14-04 -2443-vrouter-agent down in multi-node

Hi Atul,

One of the Vrouter-agent is down on multi node ubuntu-14.04 setup. Could you please check the node a5s9. There is an agent crash.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007fb3de7e0bb9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fb3de7e3fc8 in __GI_abort () at abort.c:89
#2 0x00007fb3de7d9a76 in __assert_fail_base (fmt=0x7fb3de92b370 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0xe8c7d5 "0",
    file=file@entry=0xf44d9a "controller/src/base/task.cc", line=line@entry=238, function=function@entry=0xf46000 "virtual tbb::task* TaskImpl::execute()")
    at assert.c:92
#3 0x00007fb3de7d9b22 in __GI___assert_fail (assertion=0xe8c7d5 "0", file=0xf44d9a "controller/src/base/task.cc", line=238,
    function=0xf46000 "virtual tbb::task* TaskImpl::execute()") at assert.c:101
#4 0x0000000000e5f477 in TaskImpl::execute() ()
#5 0x00007fb3df3b0b3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007fb3df3ac816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007fb3df3abf4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007fb3df3a80ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fb3df3a82f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fb3df5cc182 in start_thread (arg=0x7fb3d6c2b700) at pthread_create.c:312
#11 0x00007fb3de8a4fbd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

root@a5s9:/var/log/contrail# contrail-status
== Contrail vRouter ==
supervisor-vrouter: active
contrail-vrouter-agent failed
contrail-vrouter-nodemgr active

========Run time service failures=============
/var/crashes/core.contrail-vroute.1616.a5s9.1415379503
/var/crashes/core.contrail-vroute.6419.a5s9.1415379518
/var/crashes/core.contrail-vroute.6912.a5s9.1415379526
/var/crashes/core.contrail-vroute.5464.a5s9.1415379506
/var/crashes/core.contrail-vroute.5957.a5s9.1415379512

Testbed:
=========
host1 = 'root@10.84.14.6'
host2 = 'root@10.84.14.7'
host3 = 'root@10.84.14.8'
host4 = 'root@10.84.14.9'
host5 = 'root@10.84.14.10'

#Role definition of the hosts.
env.roledefs = {
    'all': [host1, host2, host3, host4, host5],
    'cfgm': [host1, host2],
    'openstack': [host3],
    'control': [host1, host2],
    'compute': [host4, host5],
    'collector': [host1, host2, host3],
    'webui': [host1],
    'database': [host1, host2, host3],
    'build': [host_build],
}

env.hostnames = {
    'all': ['a5s6', 'a5s7', 'a5s8','a5s9', 'a5s10']

Thanks,
Shaju

[shajuvk (shajuvk)]

logs: /cs-shared/shaju/bugs/bug-1390594/

[Atul Moghe (moghea)]

Running test_vm_with_sec_group test cases would make agent to crash.
Issue can be recreated with this particular test case..

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/4396
Committed: http://github.org/Juniper/contrail-controller/commit/3cd4beb288b9a69e0ea8212c07a639dbdee75418
Submitter: Zuul
Branch: master

commit 3cd4beb288b9a69e0ea8212c07a639dbdee75418
Author: Praveen K V <email address hidden>
Date: Sat Nov 8 23:28:07 2014 +0530

Fix exception thrown by boost in MaskToPrefix

Boost Ip4Address() takes unsigned long as argument. If the address
passed is greater than 0xFFFFFFFF, it throws exception. Method
MaskToPrefix has potential to pass values greater than 0xFFFFFFFF.

Renamed MaskToPrefix to PrefixToIpAddress and moved to base/util.h.
The new method uses uint32_t for mask. This ensures the value is
never greater than 0xFFFFFFFF

Change-Id: I64a69c59e7902142e68e09ecef8964a3d79933f7
Closes-Bug: #1390594

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/15603
Submitter: ?douard Thuleau (<email address hidden>)

[Daisuke Nakajima (dnakajima)]

this issue is seen at R2.11.1 -15
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fd6dc542cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007fd6dc542cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007fd6dc5460d8 in __GI_abort () at abort.c:89
#2 0x00007fd6dc53bb86 in __assert_fail_base (fmt=0x7fd6dc68c830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x1019f75 "0",
    file=file@entry=0x10e7951 "controller/src/base/task.cc", line=line@entry=253,
    function=function@entry=0x10e9000 "virtual tbb::task* TaskImpl::execute()") at assert.c:92
#3 0x00007fd6dc53bc32 in __GI___assert_fail (assertion=0x1019f75 "0", file=0x10e7951 "controller/src/base/task.cc", line=253,
    function=0x10e9000 "virtual tbb::task* TaskImpl::execute()") at assert.c:101
#4 0x0000000000fdd9f3 in TaskImpl::execute() ()
#5 0x00007fd6dd111b3a in ?? () from /usr/lib/libtbb.so.2
#6 0x00007fd6dd10d816 in ?? () from /usr/lib/libtbb.so.2
#7 0x00007fd6dd10cf4b in ?? () from /usr/lib/libtbb.so.2
#8 0x00007fd6dd1090ff in ?? () from /usr/lib/libtbb.so.2
#9 0x00007fd6dd1092f9 in ?? () from /usr/lib/libtbb.so.2
#10 0x00007fd6dd32d182 in start_thread (arg=0x7fd6d4fc6700) at pthread_create.c:312
#11 0x00007fd6dc60647d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/15603
Committed: http://github.org/Juniper/contrail-controller/commit/5f4ec358c5704806b1a1d0bec0043a1a517ddb2c
Submitter: Zuul
Branch: R1.10

commit 5f4ec358c5704806b1a1d0bec0043a1a517ddb2c
Author: Praveen K V <email address hidden>
Date: Sat Nov 8 23:28:07 2014 +0530

Fix exception thrown by boost in MaskToPrefix

Boost Ip4Address() takes unsigned long as argument. If the address
passed is greater than 0xFFFFFFFF, it throws exception. Method
MaskToPrefix has potential to pass values greater than 0xFFFFFFFF.

Renamed MaskToPrefix to PrefixToIpAddress and moved to base/util.h.
The new method uses uint32_t for mask. This ensures the value is
never greater than 0xFFFFFFFF

Change-Id: I64a69c59e7902142e68e09ecef8964a3d79933f7
Closes-Bug: #1390594
(cherry picked from commit 3cd4beb288b9a69e0ea8212c07a639dbdee75418)