Mir

CI test failure in multiple tests

Bug #1401364 reported by Daniel van Vugt
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mir
Fix Released
High
Mir development team
mir (Ubuntu)
Fix Released
High
Unassigned

Bug Description

Multiple tests fail in a similar fashion:

https://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/352/consoleFull

For example:

ClientLibraryThread.handles_no_signals fails due to memory leak (valgrind)

8: [ RUN ] ClientLibraryThread.handles_no_signals
8: ==10098==
8: ==10098== HEAP SUMMARY:
8: ==10098== in use at exit: 10,826 bytes in 83 blocks
8: ==10098== total heap usage: 254,821 allocs, 254,738 frees, 16,598,040 bytes allocated
8: ==10098==
8: ==10098== 92 (64 direct, 28 indirect) bytes in 1 blocks are definitely lost in loss record 63 of 83
8: ==10098== at 0x4C2B100: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
8: ==10098== by 0x5288C3C: allocate (new_allocator.h:104)
8: ==10098== by 0x5288C3C: allocate (alloc_traits.h:357)
8: ==10098== by 0x5288C3C: __shared_count<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr_base.h:616)
8: ==10098== by 0x5288C3C: __shared_ptr<std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr_base.h:1090)
8: ==10098== by 0x5288C3C: shared_ptr<std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:316)
8: ==10098== by 0x5288C3C: allocate_shared<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:588)
8: ==10098== by 0x5288C3C: make_shared<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:604)
8: ==10098== by 0x5288C3C: _M_make_routine<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (thread:196)
8: ==10098== by 0x5288C3C: thread<mir::client::rpc::StreamSocketTransport::init()::<lambda()> > (thread:138)
8: ==10098== by 0x5288C3C: mir::client::rpc::StreamSocketTransport::init() (stream_socket_transport.cpp:188)
8: ==10098== by 0x5289A62: mir::client::rpc::StreamSocketTransport::StreamSocketTransport(mir::Fd const&) (stream_socket_transport.cpp:41)
8: ==10098== by 0x527B894: mir::client::rpc::make_rpc_channel(std::string const&, std::shared_ptr<mir::client::SurfaceMap> const&, std::shared_ptr<mir::client::DisplayConfiguration> const&, std::shared_ptr<mir::client::rpc::RpcReport> const&, std::shared_ptr<mir::client::LifecycleControl> const&, std::shared_ptr<mir::client::EventSink> const&) (make_socket_rpc_channel.cpp:55)
8: ==10098== by 0x5271DB3: operator() (default_connection_configuration.cpp:92)
8: ==10098== by 0x5271DB3: std::_Function_handler<std::shared_ptr<google::protobuf::RpcChannel> (), mir::client::DefaultConnectionConfiguration::the_rpc_channel()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (functional:2025)
8: ==10098== by 0x5272648: operator() (functional:2439)
8: ==10098== by 0x5272648: operator() (cached_ptr.h:41)
8: ==10098== by 0x5272648: mir::client::DefaultConnectionConfiguration::the_rpc_channel() (default_connection_configuration.cpp:92)
8: ==10098== by 0x525DE95: MirConnection::MirConnection(mir::client::ConnectionConfiguration&) (mir_connection.cpp:108)
8: ==10098== by 0x52625EE: connect (mir_connection_api.cpp:78)
8: ==10098== by 0x52625EE: mir_connect (mir_connection_api.cpp:143)
8: ==10098== by 0x5CB7D1: ClientLibrary_creates_surface_Test::TestBody() (test_client_library.cpp:180)
8: ==10098== by 0x7AB7A2: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest.cc:2078)
8: ==10098== by 0x7AB7A2: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114)
8: ==10098== by 0x7A1403: testing::Test::Run() (gtest.cc:2151)
8: ==10098== by 0x7A1527: testing::TestInfo::Run() [clone .part.372] (gtest.cc:2326)
8: ==10098==
8: ==10098== LEAK SUMMARY:
8: ==10098== definitely lost: 64 bytes in 1 blocks
8: ==10098== indirectly lost: 28 bytes in 2 blocks
8: ==10098== possibly lost: 426 bytes in 10 blocks
8: ==10098== still reachable: 10,308 bytes in 70 blocks
8: ==10098== suppressed: 0 bytes in 0 blocks
8: ==10098== Reachable blocks (those to which a pointer was found) are not shown.
8: ==10098== To see them, rerun with: --leak-check=full --show-leak-kinds=all
8: ==10098==
8: ==10098== For counts of detected and suppressed errors, rerun with: -v
8: ==10098== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
8: /tmp/buildd/mir-0.9.0+15.04.20141125bzr2141pkg0vivid331/tests/mir_test_framework/interprocess_client_server_test.cpp:90: Failure
8: Value of: result.exit_code
8: Expected: is equal to 0
8: Actual: 1 (of type int)
8: ==10090==
8: ==10090== HEAP SUMMARY:
8: ==10090== in use at exit: 11,727 bytes in 86 blocks
8: ==10090== total heap usage: 258,445 allocs, 258,359 frees, 16,749,998 bytes allocated
8: ==10090==
8: ==10090== 92 (64 direct, 28 indirect) bytes in 1 blocks are definitely lost in loss record 64 of 86
8: ==10090== at 0x4C2B100: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
8: ==10090== by 0x5288C3C: allocate (new_allocator.h:104)
8: ==10090== by 0x5288C3C: allocate (alloc_traits.h:357)
8: ==10090== by 0x5288C3C: __shared_count<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr_base.h:616)
8: ==10090== by 0x5288C3C: __shared_ptr<std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr_base.h:1090)
8: ==10090== by 0x5288C3C: shared_ptr<std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:316)
8: ==10090== by 0x5288C3C: allocate_shared<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::allocator<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:588)
8: ==10090== by 0x5288C3C: make_shared<std::thread::_Impl<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> >, std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (shared_ptr.h:604)
8: ==10090== by 0x5288C3C: _M_make_routine<std::_Bind_simple<mir::client::rpc::StreamSocketTransport::init()::<lambda()>()> > (thread:196)
8: ==10090== by 0x5288C3C: thread<mir::client::rpc::StreamSocketTransport::init()::<lambda()> > (thread:138)
8: ==10090== by 0x5288C3C: mir::client::rpc::StreamSocketTransport::init() (stream_socket_transport.cpp:188)
8: ==10090== by 0x5289A62: mir::client::rpc::StreamSocketTransport::StreamSocketTransport(mir::Fd const&) (stream_socket_transport.cpp:41)
8: ==10090== by 0x527B894: mir::client::rpc::make_rpc_channel(std::string const&, std::shared_ptr<mir::client::SurfaceMap> const&, std::shared_ptr<mir::client::DisplayConfiguration> const&, std::shared_ptr<mir::client::rpc::RpcReport> const&, std::shared_ptr<mir::client::LifecycleControl> const&, std::shared_ptr<mir::client::EventSink> const&) (make_socket_rpc_channel.cpp:55)
8: ==10090== by 0x5271DB3: operator() (default_connection_configuration.cpp:92)
8: ==10090== by 0x5271DB3: std::_Function_handler<std::shared_ptr<google::protobuf::RpcChannel> (), mir::client::DefaultConnectionConfiguration::the_rpc_channel()::{lambda()#1}>::_M_invoke(std::_Any_data const&) (functional:2025)
8: ==10090== by 0x5272648: operator() (functional:2439)
8: ==10090== by 0x5272648: operator() (cached_ptr.h:41)
8: ==10090== by 0x5272648: mir::client::DefaultConnectionConfiguration::the_rpc_channel() (default_connection_configuration.cpp:92)
8: ==10090== by 0x525DE95: MirConnection::MirConnection(mir::client::ConnectionConfiguration&) (mir_connection.cpp:108)
8: ==10090== by 0x52625EE: connect (mir_connection_api.cpp:78)
8: ==10090== by 0x52625EE: mir_connect (mir_connection_api.cpp:143)
8: ==10090== by 0x5CB7D1: ClientLibrary_creates_surface_Test::TestBody() (test_client_library.cpp:180)
8: ==10090== by 0x7AB7A2: HandleSehExceptionsInMethodIfSupported<testing::Test, void> (gtest.cc:2078)
8: ==10090== by 0x7AB7A2: void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (gtest.cc:2114)
8: ==10090== by 0x7A1403: testing::Test::Run() (gtest.cc:2151)
8: ==10090== by 0x7A1527: testing::TestInfo::Run() [clone .part.372] (gtest.cc:2326)
8: ==10090==
8: ==10090== LEAK SUMMARY:
8: ==10090== definitely lost: 64 bytes in 1 blocks
8: ==10090== indirectly lost: 28 bytes in 2 blocks
8: ==10090== possibly lost: 1,071 bytes in 11 blocks
8: ==10090== still reachable: 10,564 bytes in 72 blocks
8: ==10090== suppressed: 0 bytes in 0 blocks
8: ==10090== Reachable blocks (those to which a pointer was found) are not shown.
8: ==10090== To see them, rerun with: --leak-check=full --show-leak-kinds=all
8: ==10090==
8: ==10090== For counts of detected and suppressed errors, rerun with: -v
8: ==10090== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
8: /tmp/buildd/mir-0.9.0+15.04.20141125bzr2141pkg0vivid331/tests/mir_test_framework/interprocess_client_server_test.cpp:153: Failure
8: Value of: result.exit_code
8: Expected: is equal to 0
8: Actual: 1 (of type int)
8: [ FAILED ] ClientLibraryThread.handles_no_signals (707 ms)

Related branches

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Weird. I can't reproduce this locally.

Changed in mir:
status: New → Triaged
milestone: none → 0.10.0
tags: added: ci-blocker
Changed in mir:
assignee: nobody → Alexandros Frantzis (afrantzis)
Changed in mir:
status: Triaged → In Progress
Revision history for this message
Alexandros Frantzis (afrantzis) wrote :

Note that this is happening in many tests not only ClientLibraryThread.*

https://jenkins.qa.ubuntu.com/job/mir-vivid-amd64-ci/352/consoleFull

It seems the memory leak is only a consequence of something else failing:

8: Value of: result.exit_code
8: Expected: is equal to 0
8: Actual: 1 (of type int)

summary: - ClientLibraryThread.handles_no_signals fails due to memory leak
- (valgrind)
+ CI test failure in multiple tests
description: updated
Revision history for this message
Alexandros Frantzis (afrantzis) wrote :

> It seems the memory leak is only a consequence of something else failing

Actually, it's not clear what is happening. I can't reproduce locally either.

Revision history for this message
Alexandros Frantzis (afrantzis) wrote :

Changed to critical as it's blocking CI.

Changed in mir:
importance: High → Critical
Changed in mir:
assignee: Alexandros Frantzis (afrantzis) → Mir development team (mir-team)
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Test case (from Alan?)

valgrind --error-exitcode=1 --trace-children=yes --leak-check=full --show-leak-kinds=definite --errors-for-leak-kinds=definite bin/mir_acceptance_tests --gtest_filter=ClientLibraryErrorsDeathTest*:ClientLibraryThread*

Revision history for this message
Chris Halse Rogers (raof) wrote :

It's a (cunningly hidden) leak in ClientLibrary.surface_create.

Cunningly hidden in that it only manifests after a call to fork(), and only in the child.
So most people running the testsuite at home won't notice, because we run each test
in a separate process by default.

CI runs everything in one process, so hits this.

To duplicate that environment, you need to run the tests in a build with
cmake -DDISABLE_GTEST_TEST_DISCOVERY=ON -DENABLE_MEMCHECK_OPTION=ON

Revision history for this message
Daniel van Vugt (vanvugt) wrote :

... and then:

make test ARGS="-V" | tee output.txt

Changed in mir:
status: In Progress → Fix Committed
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

Reverted to High priority. It don't think this ever qualified for Critical as it never affected any users, nor prevented us from working.

Changed in mir:
importance: Critical → High
Changed in mir:
status: Fix Committed → Fix Released
Revision history for this message
Daniel van Vugt (vanvugt) wrote :

mir (0.10.0+15.04.20150107.2-0ubuntu1) vivid; urgency=medium

Changed in mir (Ubuntu):
importance: Undecided → High
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers