Intermittent mir-clang-ts-vivid-amd64-build test failures

[Alexandros Frantzis]

See http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-vivid-amd64-build/ for the full job history.

A few instances:

~~~~~~~~~~~~~~~~~

http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-vivid-amd64-build/226/console

8: [ RUN ] BufferQueue/WithTwoOrMoreBuffers.stress/1
8: ==30514==ERROR: ThreadSanitizer failed to allocate 0x12000 (73728) bytes of SignalContext (errno: 12)
8: ERROR: Failed to mmap
8/9 Test #8: mir_unit_tests ....................***Failed 739.00 sec

~~~~~~~~~~~~~~~~~

http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-vivid-amd64-build/203/console

8: [ RUN ] BufferQueue/WithAnyNumberOfBuffers.compositor_release_verifies_parameter/1
8: ==23108==ERROR: ThreadSanitizer failed to allocate 0x10000 (65536) bytes of stack depot (errno: 12)
8: ERROR: Failed to mmap

[Alexandros Frantzis (afrantzis)]

The mir-clang-ts-vivid-amd64-build job now fails consistently blocking all CI/autolanding jobs.

I haven't been able to reproduce any of the failures locally (in a vivid chroot).

There seem to be 3 different failure causes:

1. Timeout while running ClientLibraryErrorsDeathTest.creating_surface_on_garbage_connection_is_fatal

    http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-vivid-amd64-build/237/consoleFull
    http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-vivid-amd64-build/241/consoleFull

2. Inexplicable failure in BufferQueue/WithTwoOrMoreBuffers.stress/0

    http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-vivid-amd64-build/238/consoleFull
    http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-vivid-amd64-build/240/consoleFull

3. Jenkins infrastructural problem

    http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-vivid-amd64-build/239/consoleFull
    http://s-jenkins.ubuntu-ci:8080/job/mir-clang-ts-vivid-amd64-build/242/consoleFull

[Alan Griffiths (alan-griffiths)]

OK, having built this on a vivid+(bits of overlay) laptop...

I can get a crash in ClientLibraryErrorsDeathTest.creating_surface_on_garbage_connection_is_fatal by running it repeatedly.

The apparent error is:

[ RUN ] BufferQueue/WithThreeOrMoreBuffers.slow_client_framerate_matches_compositor/0
/home/alan/display_server/mir/tests/unit-tests/compositor/test_buffer_queue.cpp:1028: Failure
Value of: client_frames
Expected: is > 80
  Actual: 65 (of type unsigned long)

Program received signal SIGSEGV, Segmentation fault.
0x00000000040618c0 in testing::UnitTest::AddTestPartResult(testing::TestPartResult::Type, char const*, int, std::string const&, std::string const&) ()
(gdb) bt
#0 0x00000000040618c0 in testing::UnitTest::AddTestPartResult(testing::TestPartResult::Type, char const*, int, std::string const&, std::string const&) ()
#1 0x0000000004060fec in testing::internal::AssertHelper::operator=(testing::Message const&) const ()
#2 0x000000000224f7d7 in WithThreeOrMoreBuffers_slow_client_framerate_matches_compositor_Test::TestBody() (this=0x7d540000fa00)
    at /home/alan/display_server/mir/tests/unit-tests/compositor/test_buffer_queue.cpp:1028
#3 0x00000000040c5c01 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
#4 0x00000000040a0742 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) ()
#5 0x0000000004071a67 in testing::Test::Run() ()
#6 0x0000000004073678 in testing::TestInfo::Run() ()
#7 0x00000000040747fe in testing::TestCase::Run() ()
#8 0x0000000004082c7d in testing::internal::UnitTestImpl::RunAllTests() ()
#9 0x00000000040bc941 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#10 0x00000000040a57d8 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) ()
#11 0x000000000408254a in testing::UnitTest::Run() ()
#12 0x0000000003d17d5e in RUN_ALL_TESTS() () at /usr/include/gtest/gtest.h:2288
#13 0x0000000003d17d29 in mir_test_framework::main(int, char**) (argc=1, argv=0x7fffffffdc58)
    at /home/alan/display_server/mir/tests/mir_test_framework/command_line_server_configuration.cpp:59
#14 0x0000000003d04d90 in main(int, char**) (argc=5, argv=0x7fffffffdc58)
    at /home/alan/display_server/mir/tests/mir_test_framework/main.cpp:29

which is similar to what we see when gtest tries to log errors when the test is partially torn down. (Usually because the error is logged from a secondary thread.)

(gdb) info threads
  Id Target Id Frame
  2 Thread 0x7fffeedff700 (LWP 1596) "mir_unit_tests." 0x00007ffff272dc4d in nanosleep ()
    at ../sysdeps/unix/syscall-template.S:81
* 1 Thread 0x7ffff7faaac0 (LWP 1590) "mir_unit_tests." 0x00000000040618c0 in testing::UnitTest::AddTestPartResult(testing::TestPartResult::Type, char const*, int, std::string const&, ...

[Daniel van Vugt (vanvugt)]

Dropped severity. I haven't seen this blocking anything recently.

[Alexandros Frantzis (afrantzis)]

> Dropped severity. I haven't seen this blocking anything recently.

That's because we disabled the job last week :)

[Daniel van Vugt (vanvugt)]

No news, it seems.

[Launchpad Janitor (janitor)]

[Expired for mir (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for Mir because there has been no activity for 60 days.]