replicaset: mongodb crashes during test

[Dave Cheney]

Replica set tests crash mongodb. This blocks CI and is potentially a blocker for customers deploying juju.

[LOG] 18.03164 DEBUG juju.replicaset Set() changing replica set
from {Name: juju, Version: 3, Members: {Member{1 "localhost:39813" map[]}, Member{2 "localhost:52245" map[key0:val0]}, Member{3 "localhost:45350" map[key1:val1]}}}
  to {Name: juju, Version: 4, Members: {Member{4 "localhost:40956" map[key2:val2]}, Member{3 "localhost:45350" map[key1:val1]}, Member{1 "localhost:39813" map[]}, Member{5 "localhost:37287" map[key3:val3]}}}
[LOG] 18.04423 DEBUG juju.replicaset got EOF while running Set(), calling session.Refresh()
[LOG] 18.04434 DEBUG juju.testing tls.Dial(127.0.0.1:39813) failed with dial tcp 127.0.0.1:39813: connection refused
[LOG] 18.04518 ERROR juju mongodb has exited without being killed
[LOG] 18.04521 ERROR juju mongod: [121 lines omitted]
[LOG] 18.04521 ERROR juju mongod: Fri Apr 11 10:40:39.167 Backtrace:
[LOG] 18.04522 ERROR juju mongod: 0xb6ca63 0x6e3c00 0x7f8e7de7f4a0 0x7f8e7de7f425 0x7f8e7de82b8b 0xb31a18 0xb5b335 0x7242ad 0x7245f8 0xa00cba 0xa10bc2 0xa110a0 0xb392dc 0xb35265 0xb35bdb 0x7f8e7f31bce9 0x7f8e7fb68e9a 0x7f8e7df3d3fd
[LOG] 18.04523 ERROR juju mongod: mongod(_ZN5mongo15printStackTraceERSo+0x23) [0xb6ca63]
[LOG] 18.04523 ERROR juju mongod: mongod(_ZN5mongo10abruptQuitEi+0x350) [0x6e3c00]
[LOG] 18.04524 ERROR juju mongod: /lib/x86_64-linux-gnu/libc.so.6(+0x364a0) [0x7f8e7de7f4a0]
[LOG] 18.04525 ERROR juju mongod: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35) [0x7f8e7de7f425]
[LOG] 18.04525 ERROR juju mongod: /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b) [0x7f8e7de82b8b]
[LOG] 18.04526 ERROR juju mongod: mongod(_ZN5mongo13fassertFailedEi+0x88) [0xb31a18]
[LOG] 18.04526 ERROR juju mongod: mongod(_ZN5mongo6Socket6secureEPNS_10SSLManagerE+0x65) [0xb5b335]
[LOG] 18.04527 ERROR juju mongod: mongod(_ZN5mongo18DBClientConnection8_connectERSs+0x32d) [0x7242ad]
[LOG] 18.04528 ERROR juju mongod: mongod(_ZN5mongo18DBClientConnection7connectERKNS_11HostAndPortERSs+0x68) [0x7245f8]
[LOG] 18.04529 ERROR juju mongod: mongod(_ZN5mongo10ScopedConn7connectEv+0x5a) [0xa00cba]
[LOG] 18.04529 ERROR juju mongod: mongod(_ZN5mongo21ReplSetHealthPollTask17_requestHeartbeatERNS_13HeartbeatInfoERNS_7BSONObjERi+0x492) [0xa10bc2]
[LOG] 18.04530 ERROR juju mongod: mongod(_ZN5mongo21ReplSetHealthPollTask6doWorkEv+0x90) [0xa110a0]
[LOG] 18.04531 ERROR juju mongod: mongod(_ZN5mongo4task4Task3runEv+0x2c) [0xb392dc]
[LOG] 18.04531 ERROR juju mongod: mongod(_ZN5mongo13BackgroundJob7jobBodyEN5boost10shared_ptrINS0_9JobStatusEEE+0xb5) [0xb35265]
[LOG] 18.04532 ERROR juju mongod: mongod(_ZN5boost6detail11thread_dataINS_3_bi6bind_tIvNS_4_mfi3mf1IvN5mongo13BackgroundJobENS_10shared_ptrINS7_9JobStatusEEEEENS2_5list2INS2_5valueIPS7_EENSD_ISA_EEEEEEE3runEv+0x7b) [0xb35bdb]
[LOG] 18.04534 ERROR juju mongod: /usr/lib/libboost_thread.so.1.46.1(thread_proxy+0x69) [0x7f8e7f31bce9]
[LOG] 18.04534 ERROR juju mongod: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a) [0x7f8e7fb68e9a]
[LOG] 18.04534 ERROR juju mongod: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8e7df3d3fd]

[Tim Penhey (thumper)]

relevant bits pushed through c++filt to demangle function names:

[LOG] 18.04526 ERROR juju mongod: mongod(mongo::fassertFailed(int)+0x88) [0xb31a18]
[LOG] 18.04526 ERROR juju mongod: mongod(mongo::Socket::secure(mongo::SSLManager*)+0x65) [0xb5b335]
[LOG] 18.04527 ERROR juju mongod: mongod(mongo::DBClientConnection::_connect(std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x32d) [0x7242ad]
[LOG] 18.04528 ERROR juju mongod: mongod(mongo::DBClientConnection::connect(mongo::HostAndPort const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)+0x68) [0x7245f8]
[LOG] 18.04529 ERROR juju mongod: mongod(mongo::ScopedConn::connect()+0x5a) [0xa00cba]
[LOG] 18.04529 ERROR juju mongod: mongod(mongo::ReplSetHealthPollTask::_requestHeartbeat(mongo::HeartbeatInfo&, mongo::BSONObj&, int&)+0x492) [0xa10bc2]
[LOG] 18.04530 ERROR juju mongod: mongod(mongo::ReplSetHealthPollTask::doWork()+0x90) [0xa110a0]
[LOG] 18.04531 ERROR juju mongod: mongod(mongo::task::Task::run()+0x2c) [0xb392dc]
[LOG] 18.04531 ERROR juju mongod: mongod(mongo::BackgroundJob::jobBody(boost::shared_ptr<mongo::BackgroundJob::JobStatus>)+0xb5) [0xb35265]
[LOG] 18.04532 ERROR juju mongod: mongod(boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf1<void, mongo::BackgroundJob, boost::shared_ptr<mongo::BackgroundJob::JobStatus> >, boost::_bi::list2<boost::_bi::value<mongo::BackgroundJob*>, boost::_bi::value<boost::shared_ptr<mongo::BackgroundJob::JobStatus> > > > >::run()+0x7b) [0xb35bdb]

[Curtis Hovey (sinzui)]

I have published 1:2.4.6-0ubuntu5~ubuntu12.04.1~juju1 to the juju stable ppa. This failing test still fails about every 15 runs.

[Ian Booth (wallyworld)]

Since the recent improvements to replicaset tests, I don't think this one comes up anymore. So marking as incomplete. We can reopen if it reoccurs.

[Launchpad Janitor (janitor)]

[Expired for juju-core because there has been no activity for 60 days.]