contrail-control, contrail-collector and contrail-dns core with backtrace complaining of missing file sigandset.c

[wenqing liang]

R4.1.0.0-46 newton full ha vm cluster of 1604.

(gdb) bt
#0 0x00007f3329870428 in sigandset (dest=0x15ad, left=0x7982, right=0x6) at sigandset.c:33
#1 0x0000000000000020 in ?? ()
#2 0x0000000000000000 in ?? ()
(gdb)

core.contrail-contro.5549.server6.1504716969 uploaded to /cs-shared/bugs/

[Ananth Suryanarayana (anantha-l)]

I don't find the core file

buntu-build02 /build/anantha/mvpn_commit> cd /cs-shared/bugs/*1715563
zsh: no matches found: /cs-shared/bugs/*1715563
ubuntu-build02 /build/anantha/mvpn_commit> cd /cs-shared/bugs/*1715563*
zsh: no matches found: /cs-shared/bugs/*1715563*
ubuntu-build02 /build/anantha/mvpn_commit> cd /cs-shared/bugs/core.contrail-contro.5549.server6.1504716969
/users/anantha/bin/mycd.sh:cd:6: no such file or directory: /tmp/anantha#_priv_cd.link
ubuntu-build02 /build/anantha/mvpn_commit> cd /cs-shared/bugs/*core.contrail-contro.5549.server6.1504716969*
zsh: no matches found: /cs-shared/bugs/*core.contrail-contro.5549.server6.1504716969*
ubuntu-build02 /build/anantha/mvpn_commit> ls /cs-shared/bugs/*core*
zsh: no matches found: /cs-shared/bugs/*core*
ubuntu-build02 /build/anantha/mvpn_commit>

[Sudheendra Rao (sudheendra-k)]

core files are copied to:

[sudheendra@nodem4 1715563]$ pwd
/cs-shared/bugs/1715563
[sudheendra@nodem4 1715563]$ ls
core.contrail-collec.3446.nodem16.1509924757 core.contrail-contro.6837.nodem16.1509924765 core.contrail-dns.7669.nodem16.1509924771
[sudheendra@nodem4 1715563]$

[Ritam Gangopadhyay (ritam)]

bt :-
Core was generated by `/usr/bin/contrail-control'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f954d483428 in sigandset (dest=0x1e31, left=0x1e31, right=0x6) at sigandset.c:33
33 sigandset.c: No such file or directory.
(gdb) bt
#0 0x00007f954d483428 in sigandset (dest=0x1e31, left=0x1e31, right=0x6) at sigandset.c:33
#1 0x0000000000000020 in ?? ()
#2 0x0000000000000000 in ?? ()
(gdb) q

Topology :
DISTRO : "Ubuntu 16.04.2 LTS"
SKU : ocata
Config Nodes : [u'nodem16', u'nodem17', u'nodem18']
Control Nodes : [u'nodem16', u'nodem17', u'nodem18']
Comp! ute Nodes : [u'nodem19', u'nodem20']
Openstack Node : [u'nodem16',! u'nodem17', u'nodem18']
WebUI Node : [u'nodem16', u'nodem17', u'nodem18']
Analytics Nodes : [u'nodem16', u'nodem17', u'nodem18']
Database Nodes : [u'nodem16', u'nodem17', u'nodem18']
Physical Devices : [u'blr-mx2', u"'blr-mx2'"]
LB Nodes : [u'nodea10']

[Ananth Suryanarayana (anantha-l)]

Can you please provide a setup to debug. I could not extract get any meaningful stack trace from the core -- thanks

[Ritam Gangopadhyay (ritam)]

Setup provided above in comment #3.

[Ananth Suryanarayana (anantha-l)]

(gdb) bt
#0 0x00007ffff52ba428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff52bc02a in __GI_abort () at abort.c:89
#2 0x00007ffff52b2bd7 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x10ba42f "valid_", file=file@entry=0x10ba568 "/usr/include/boost/token_iterator.hpp", line=line@entry=56,
    function=function@entry=0x1199520 <_ZZNK5boost14token_iteratorINS_14char_separatorIcSt11char_traitsIcEEEN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcS3_SaIcEEEEESC_E11dereferenceEvE19__PRETTY_FUNCTION__> "const Type& boost::token_iterator<TokenizerFunc, Iterator, Type>::dereference() const [with TokenizerFunc = boost::char_separator<char, std::char_traits<char> >; Iterator = __gnu_cxx::__normal_iterato"...) at assert.c:92
#3 0x00007ffff52b2c82 in __GI___assert_fail (assertion=assertion@entry=0x10ba42f "valid_", file=file@entry=0x10ba568 "/usr/include/boost/token_iterator.hpp", line=line@entry=56,
    function=function@entry=0x1199520 <_ZZNK5boost14token_iteratorINS_14char_separatorIcSt11char_traitsIcEEEN9__gnu_cxx17__normal_iteratorIPKcNSt7__cxx1112basic_stringIcS3_SaIcEEEEESC_E11dereferenceEvE19__PRETTY_FUNCTION__> "const Type& boost::token_iterator<TokenizerFunc, Iterator, Type>::dereference() const [with TokenizerFunc = boost::char_separator<char, std::char_traits<char> >; Iterator = __gnu_cxx::__normal_iterato"...) at assert.c:101
#4 0x0000000000d11cb2 in boost::token_iterator<boost::char_separator<char, std::char_traits<char> >, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::dereference (this=<optimized out>) at /usr/include/boost/token_iterator.hpp:56
#5 boost::iterators::iterator_core_access::dereference<boost::token_iterator<boost::char_separator<char, std::char_traits<char> >, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > (f=...) at /usr/include/boost/iterator/iterator_facade.hpp:540
#6 boost::iterators::detail::iterator_facade_base<boost::token_iterator<boost::char_separator<char, std::char_traits<char> >, __gnu_cxx::__normal_iterator<char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::iterators::forward_traversal_tag, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, false, false>::operator* (this=<optimized out>) at /usr/include/boost/iterator/iterator_facade.hpp:646
#7 ConfigAmqpClient::ConfigAmqpClient (this=0x17f5090, mgr=<optimized out>, hostname=..., module_name=..., options=...) at controller/src/config/config-client-mgr/config_amqp_client.cc:86
#8 0x0000000000d1d424 in ConfigClientManager::SetUp (this=this@entry=0x17e94e0) at controller/src/config/config-client-mgr/config_...

[Ananth Suryanarayana (anantha-l)]

Provisioning issue. Can you please reassign -- thanks

[Abhay Joshi (abhayj)]

Sachin,

Can you please get someone to look into this - this is to do with internal sensible.

Thanks,

Abhay

[Nagendra Prasath (npchandran)]

Need more info:
> attach sm json files used
> attach /etc/contrailctl
> attach /etc/contrail/* from all containers

[wenqing liang (wliang)]

Not seen again and hence reducing the severity.

[Jeba Paulaiyan (jebap)]

Not seen now. Pls re-open if it happens again