tst_slotslayout is flaky

[Tim Peeters]

I got a failure on CI with tst_slotslayout:

Executing dbus-test-runner --task gdb -p --quiet -n tst_slotslayout.qml -m 500 -p --batch -p -ex -p 'set print thread-events off' -p -ex -p run -p -ex -p bt -p --return-child-result -p --args -p /tmp/buildd/ubuntu-ui-toolkit-gles-1.3.1872+16.04.20160308/tests/unit_x11/tst_components/tst_components -p -input -p tst_slotslayout.qml -p -maxwarnings -p 100 -p -o -p /tmp/buildd/ubuntu-ui-toolkit-gles-1.3.1872+16.04.20160308/tests/test_tst_slotslayout.qml.xml,xunitxml -p -o -p -,txt
Working directory: /tmp/buildd/ubuntu-ui-toolkit-gles-1.3.1872+16.04.20160308/tests/unit_x11/tst_components
DBus daemon: unix:abstract=/tmp/dbus-EJFObKuEFm,guid=d5fc837e49f55972e9b2c748571133ef
tst_slotslayout.qml: Started with PID: 28926
Undefined command: "". Try "help".
tst_slotslayout.qml: [Thread debugging using libthread_db enabled]
tst_slotslayout.qml: Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1".
tst_slotslayout.qml: [New Thread 0xee5f3b40 (LWP 28938)]
tst_slotslayout.qml: [New Thread 0xecd9eb40 (LWP 28939)]
tst_slotslayout.qml: ********* Start testing of components *********
tst_slotslayout.qml: Config: Using QtTest library 5.4.1, Qt 5.4.1 (i386-little_endian-ilp32 shared (dynamic) release build; by GCC 4.9.2)
tst_slotslayout.qml: PASS : components::SlotsLayout::initTestCase()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_changeAttachedPropertiesInMainSlot()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_changeSlotPosition()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_contentPadding(Empty)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_contentPadding(Labels)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_contentPadding(One leading slot)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_contentPadding(One trailing slot)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_contentPadding(One leading, one trailing slots)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_contentPadding(Two trailing)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_contentPadding(One leading, two trailing slots)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_contentPadding(One leading, two trailing slots with custom paddings)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_customPadding(Custom padding)
tst_slotslayout.qml: INFO : components::SlotsLayout::test_defaultLabelsQmlContext() Did not receive message: "file:///tmp/buildd/ubuntu-ui-toolkit-gles-1.3.1872+16.04.20160308/tests/unit_x11/tst_components/tst_slotslayout.qml: QML Label: Cannot open: file:///test.png"
tst_slotslayout.qml: FAIL! : components::SlotsLayout::test_defaultLabelsQmlContext() Not all expected messages were received
tst_slotslayout.qml: QWARN : components::SlotsLayout::test_defaultLabelsQmlContext() file:///tmp/buildd/ubuntu-ui-toolkit-gles-1.3.1872+16.04.20160308/tests/unit_x11/tst_components/tst_slotslayout.qml: QML Label: Cannot open: file:///test.png
tst_slotslayout.qml: PASS : components::SlotsLayout::test_defaultLabelsWrappingAndElide()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_defaultMainSlotHeight()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_defaultSlotsAttachedProps()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_fontsizeChange()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_ignoreSomeChangesOnMainSlot()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_implicitSize(Empty)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_implicitSize(Labels)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_implicitSize(One leading slot)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_implicitSize(One trailing slot)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_implicitSize(One leading, one trailing slots)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_implicitSize(Two trailing)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_implicitSize(One leading, two trailing slots)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_implicitSize(One leading, two trailing slots with custom paddings)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_layout(Empty)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_layout(Labels)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_layout(One leading slot)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_layout(One trailing slot)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_layout(One leading, one trailing slots)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_layout(Two trailing)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_layout(One leading, two trailing slots)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_layout(One leading, two trailing slots with custom paddings)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_mainSlotSize()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_multilineLabelsPositioning()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_nullSizeSlots()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_overrideVerticalPositioning()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_relativePositioning(Relative positioning 1)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_relativePositioning(Relative positioning 2)
tst_slotslayout.qml: PASS : components::SlotsLayout::test_relayoutAfterChangingSlotsSize()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_slotVisibilityChange()
tst_slotslayout.qml: PASS : components::SlotsLayout::test_warningOnAttemptToChangeListItemLayoutMainSlot()
tst_slotslayout.qml: PASS : components::SlotsLayout::cleanupTestCase()
tst_slotslayout.qml: Totals: 43 passed, 1 failed, 0 skipped, 0 blacklisted
tst_slotslayout.qml: ********* Finished testing of components *********
tst_slotslayout.qml: [Thread 0xecd9eb40 (LWP 28939) exited]
tst_slotslayout.qml: [Thread 0xee5f3b40 (LWP 28938) exited]
tst_slotslayout.qml: [Inferior 1 (process 28929) exited with code 01]
No stack.
tst_slotslayout.qml: Exited with status 256
tst_slotslayout.qml: Shutting down
DBus daemon: Shutdown
tst_slotslayout.qml exited with 0

I did not see it before, so I assume the test is flaky. The full CI run where it failed is here https://jenkins.ubuntu.com/ubuntu-sdk/job/ubuntu-ui-toolkit-ci-i386-gles-stable/457/console

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andrea Bernabei (faenil)]

I guess what happens here is
waitForRendering(obj) is not enough to wait for the warning to be output.

I guessed that would have been enough, since the labels' text is set in initialization phase, and not as a late assignment.

The Image is loaded async by default (afaik), so maybe what happens is the text is first rendered without the image and then the image finishes its loading process (with a failure, as the file does not exist) and outputs the warning. But at that point the test could already be finished, as the first render was enough to let it through "waitForRendering".

Because of that, I guess what happens is the warning is output after destroy() is called (that just schedules the destruction of the item, it does not do it immediately), i.e. when the test is already finished. As a consequence, the test fails.

I have just pushed an MR as a tentative fix, as I could not reproduce the issue locally, not even playing with cpulimit :(

[Andrea Bernabei (faenil)]

I ran the full tst_slotslayout suite locally for 4470 times WITHOUT the fix (using "xvfb.sh qmltestrunner -input tst_slotslayout -import <path_to_uitk_build/qml>", and could not get a single failure.

I can't reproduce it :/

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-ui-toolkit - 1.3.1988+16.10.20160601

---------------
ubuntu-ui-toolkit (1.3.1988+16.10.20160601) yakkety; urgency=medium

  [ Christian Dywan ]
  * UCAbstractButton needs to accept key release events. Fixes LP: #1581026

  [ Andrea Bernabei ]
  * Documentation: fix missing or outdated imports in QML modules (Popups,
    Pickers, Themes, ListItems), misuse of qdoc API and similar.
    Fixes LP: #1570156, LP: #1585557, LP: #1585585, LP: #1585591.
  * tst_slotslayout: tentative fix for a very rare CI failure.
    Fixes LP: #1571426

  [ Tarmac, Zoltán Balogh ]
  * OTA12-2016-06-01

 -- Zoltan Balogh <email address hidden> Wed, 01 Jun 2016 11:52:35 +0000