CI segfault in mir-mediumtests-runner-mako after AndroidHardwareSanity tests

[Alexandros Frantzis]

As seen in https://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-mako/6863/consoleFull

...
[ RUN ] AndroidHardwareSanity.can_allocate_hw_buffer
[ OK ] AndroidHardwareSanity.can_allocate_hw_buffer (0 ms)
/bin/bash: line 1: 6096 Segmentation fault (core dumped) umockdev-run -- mir_integration_tests
+ mir_rc=-1

Perhaps a bug in the teardown of the AndroidHardwareSanity tests?

[Alan Griffiths (alan-griffiths)]

This on krillin

https://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7014/console
...
[----------] 6 tests from AndroidHardwareSanity
[1444833079.254946] mircommon: Loading modules from: /usr/lib/arm-linux-gnueabihf/mir/server-platform
[1444833079.255444] mircommon: Loading module: /usr/lib/arm-linux-gnueabihf/mir/server-platform/graphics-android.so.6
[1444833079.255588] mircommon: Loading module: /usr/lib/arm-linux-gnueabihf/mir/server-platform/graphics-mesa-kms.so.6
[1444833079.255672] mircommon: Loading module: /usr/lib/arm-linux-gnueabihf/mir/server-platform/server-mesa-x11.so.6
[1444833079.261257] mircommon: Loading module: /usr/lib/arm-linux-gnueabihf/mir/server-platform/graphics-android.so.5
[1444833079.265158] mircommon: Loading module: /usr/lib/arm-linux-gnueabihf/mir/server-platform/input-evdev.so.3
[1444833079.269437] mircommon: Loading module: /usr/lib/arm-linux-gnueabihf/mir/server-platform/input-stub.so
[1444833079.269596] mircommon: Loading module: /usr/lib/arm-linux-gnueabihf/mir/server-platform/graphics-dummy.so
[1444833079.270505] mirplatform: Found graphics driver: android
[1444833079.271059] mirplatform: Found graphics driver: mesa-kms
[1444833079.271188] mirplatform: Found graphics driver: mesa-x11
[1444833079.271513] mirplatform: Found graphics driver: dummy
[1444833079.272475] mirserver: Selected driver: android (version 0.17.0)
[1444833079.305840] mirserver: Initial display configuration:
[1444833079.306382] mirserver: 0.0: LVDS 4.6" 57x101mm
[1444833079.306479] mirserver: Current mode 540x960 66.57Hz
[1444833079.306572] mirserver: Preferred mode 540x960 66.57Hz
[1444833079.306681] mirserver: Logical position +0+0
[1444833079.306827] mirserver: 0.1: unused DisplayPort
[1444833079.309314] mirserver: Mir version 0.17.0
[ RUN ] AndroidHardwareSanity.client_can_draw_with_cpu
[ OK ] AndroidHardwareSanity.client_can_draw_with_cpu (30 ms)
[ RUN ] AndroidHardwareSanity.client_can_draw_with_gpu
[ OK ] AndroidHardwareSanity.client_can_draw_with_gpu (31 ms)
[ RUN ] AndroidHardwareSanity.display_can_post
[ OK ] AndroidHardwareSanity.display_can_post (6 ms)
[ RUN ] AndroidHardwareSanity.display_can_post_overlay
[ OK ] AndroidHardwareSanity.display_can_post_overlay (2 ms)
[ RUN ] AndroidHardwareSanity.can_allocate_sw_buffer
[ OK ] AndroidHardwareSanity.can_allocate_sw_buffer (3 ms)
[ RUN ] AndroidHardwareSanity.can_allocate_hw_buffer
[ OK ] AndroidHardwareSanity.can_allocate_hw_buffer (0 ms)
/bin/bash: line 1: 3751 Segmentation fault umockdev-run -- mir_integration_tests

[Alan Griffiths (alan-griffiths)]

*Possibly* related to another segfault on krillin: lp:1506137

[Alan Griffiths (alan-griffiths)]

Nearest I have got to reproducing:

Works: $ gdb --args bin/mir_integration_tests --gtest_repeat=8 --gtest_filter=AndroidHardwareSanity.*:-AndroidHardwareSanity.display_can_post_overlay
Fails: $ gdb --args bin/mir_integration_tests --gtest_repeat=2 --gtest_filter=AndroidHardwareSanity.display_can_post_overlay

Stack trace is useless

[Alan Griffiths (alan-griffiths)]

OTOH repeating a test that sets up and tears down the drivers "can be problematic". So this may be unrelated.

[Alan Griffiths (alan-griffiths)]

So, avoiding the trap of reloading the drivers in a single process, I can get a (useless) core in a minute or so with:

    $ while bin/mir_integration_tests --gtest_filter=AndroidHardwareSanity.can_allocate_hw_buffer; do :; done

Interestingly, the following doesn't core (for a significantly longer time):

    $ while bin/mir_integration_tests --gtest_filter=AndroidHardwareSanity.*:-AndroidHardwareSanity.can_allocate_hw_buffer; do :; done

[Alan Griffiths (alan-griffiths)]

I can't reproduce on mako, and the linked workaround "fixes" krillin. After looking at http://paste.ubuntu.com/12789072/ Kevin thinks that the problem is in the (closed source) driver so I don't think we can trace any further.

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:mir at revision None, scheduled for release in mir, milestone 0.18.0

[Alan Griffiths (alan-griffiths)]

Leaving open as we have a workaround, not a fix

[Alexandros Frantzis (afrantzis)]

Two more instances:

https://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7164/consoleFull
https://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7165/consoleFull

[Alexandros Frantzis (afrantzis)]

https://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7169/

[Alexandros Frantzis (afrantzis)]

We are still seeing this on a regular basis, raising priority.

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:mir at revision None, scheduled for release in mir, milestone 0.18.0

[Alexandros Frantzis (afrantzis)]

Seen again: https://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7582/consoleFull

[Alexandros Frantzis (afrantzis)]

> Seen again: https://jenkins.qa.ubuntu.com/job/mir-mediumtests-runner-touch/7582/consoleFull

Actually, this seem to a be problem with the changes introduced in that particular branch, trunk seems to be fine.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mir - 0.18.0+16.04.20151216.1-0ubuntu1

---------------
mir (0.18.0+16.04.20151216.1-0ubuntu1) xenial; urgency=medium

  [ Kevin DuBois ]
  * New upstream release 0.18.0 (https://launchpad.net/mir/+milestone/0.18.0)
    - ABI summary: Only servers need rebuilding;
      . Mirclient ABI unchanged at 9
      . Mirserver ABI bumped to 36
      . Mircommon ABI unchanged at 5
      . Mirplatform ABI unchanged at 11
      . Mirprotobuf ABI unchanged at 3
      . Mirplatformgraphics ABI bumped to 7
      . Mirclientplatform ABI unchanged at 3
      . Mirinputplatform ABI added. Current version is 4
    - Enhancements:
      . Use libinput by default, and remove the android input stack
      . Add x11 input probing
      . Add alternative buffer swapping mechanism internally, available with
        --nbuffers 0
      . Automatic searching and selection of input platforms
      . Better support for themed cursors
      . Add demo client that uses multiple buffer streams in one surface
      . Improve fingerpaint demo to use touch pressure
      . Allow for configuring cursor acceleration, scroll speed and left or
        right handed mice
      . Allow for setting a base display configuration via client api
      . Various nested server multimonitor fixes and stability improvements
      . Remove DepthId from the SurfaceStack
    - Bug fixes:
      . Unit test failures in Display.* on Android (LP: #1519276)
      . Build failure due to missing dependency of client rpc code on mir
        protobuf (LP: #1518372)
      . Test failure in
        NestedServer.display_configuration_reset_when_application_exits
        (LP: #1517990)
      . CI test failures in various NesterServer tests (LP: #1517781)
      . FTBFS with -DMIR_PLATFORM=android (LP: #1517532)
      . Nesting Mir servers with assorted display configs causes lockup
        (LP: #1516670)
      . [testsfail] RaiseSurfaces.motion_events_dont_prevent_raise
        (LP: #1515931)
      . CI test failures in GLMark2Test (LP: #1515660)
      . Shells that inject user input events need to agree with the system
        compositor on the clock to use (LP: #1515515)
      . mircookie-dev is missing nettle-dev dependency (LP: #1514391)
      . Segmentation fault on server shutdown with mesa-kms (LP: #1513901)
      . mircookie requires nettle but libmircookie-dev doesn't depend on it
        (LP: #1513792)
      . libmircookie1 package does not list libnettle as dependency
        (LP: #1513225)
      . display configuration not reset when application exits (LP: #1511798)
      . unplugging external monitor causes nested server to throttle client
        (LP: #1511723)
      . 1/2 screen on external monitor (LP: #1511538)
      . unity-system-compositor crash, no interaction on windowed mode
        (LP: #1511095)
      . [regression] arm64/powerpc cross compile doesn't build any more
        (LP: #1510778)
      . mir_connection_get_egl_pixel_format() crashes if libEGL is loaded
        RTLD_LAZY (LP: #1510218)
      . [multimonitor] nested server surface positioning incorrect
        (LP: #1506846)
      . unity-system-compositor fails to build against lp:mir r3027
   ...