Agent crash in <Interface::GetOsParams(Agent*)::__PRETTY_FUNCTION__>

[amit surana]

contrail 2.21.1 #15

core file location: 10.84.5.112:/cs-shared/bugs/1531560/core*.1531560

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/contrail-vrouter-agent'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f7531b1bcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0 0x00007f7531b1bcc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f7531b1f0d8 in __GI_abort () at abort.c:89
#2 0x00007f7531b14b86 in __assert_fail_base (fmt=0x7f7531c65830 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
    assertion=assertion@entry=0x100e6ae "fd >= 0", file=file@entry=0x100e6b8 "controller/src/vnsw/agent/oper/interface.cc",
    line=line@entry=432,
    function=function@entry=0x100e980 <Interface::GetOsParams(Agent*)::__PRETTY_FUNCTION__> "virtual void Interface::GetOsParams(Agent*)")
    at assert.c:92
#3 0x00007f7531b14c32 in __GI___assert_fail (assertion=0x100e6ae "fd >= 0", file=0x100e6b8 "controller/src/vnsw/agent/oper/interface.cc",
    line=432, function=0x100e980 <Interface::GetOsParams(Agent*)::__PRETTY_FUNCTION__> "virtual void Interface::GetOsParams(Agent*)")
    at assert.c:101
#4 0x000000000095466a in Interface::GetOsParams (this=0x7f7500000c10, agent=<optimized out>)
    at controller/src/vnsw/agent/oper/interface.cc:432
#5 0x00000000009b8a48 in VmInterfaceOsOperStateData::OnResync (this=<optimized out>, table=<optimized out>, vmi=0x7f7500000c10,
    force_update=<optimized out>) at controller/src/vnsw/agent/oper/vm_interface.cc:1995
#6 0x00000000009c5d3c in VmInterface::Resync (this=0x7f7500000c10, table=<optimized out>, data=0x7f7518140630)
    at controller/src/vnsw/agent/oper/vm_interface.cc:1123
#7 0x00000000009553ba in AgentOperDBTable::Resync (this=0x7f75200167f0, entry=0x7f7500000c20, req=0x7f7518077200)
    at controller/src/vnsw/agent/oper/oper_db.h:175
#8 0x00000000009f043c in AgentDBTable::Input (this=0x7f75200167f0, partition=0x7f7520016aa0, client=<optimized out>, req=0x7f7518077200)
    at controller/src/vnsw/agent/cmn/agent_db.cc:119
#9 0x0000000000edeca6 in DBPartition::QueueRunner::Run (this=0x7f751813f420) at controller/src/db/db_partition.cc:187
#10 0x0000000000fdd8b0 in TaskImpl::execute (this=0x7f75347dfc40) at controller/src/base/task.cc:238
#11 0x00007f75326eab3a in ?? () from /usr/lib/libtbb.so.2
#12 0x00007f75326e6816 in ?? () from /usr/lib/libtbb.so.2
#13 0x00007f75326e5f4b in ?? () from /usr/lib/libtbb.so.2
#14 0x00007f75326e20ff in ?? () from /usr/lib/libtbb.so.2
#15 0x00007f75326e22f9 in ?? () from /usr/lib/libtbb.so.2
#16 0x00007f7532906182 in start_thread (arg=0x7f7528d99700) at pthread_create.c:312
#17 0x00007f7531bdf47d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
(gdb)

[Hari Prasad Killi (haripk)]

int fd = socket(AF_LOCAL, SOCK_STREAM, 0);
assert(fd >= 0);

Looks like the max fd limit is reached. Need to check the limit configured and what fds are used. Please update with the test being done when this was seen and if it can be reproduced.