Flaky gallery.test_textinput.TextInputTests.test_textfield_numbers, fix or disable

[Timo Jyrinki]

We're hunting for the remaining flaky (unstable) tests. This one:

http://ci.ubuntu.com/smokeng/trusty/touch/maguro/55:20131209.1:20131203/5355/ubuntu-ui-toolkit-autopilot/559350/

fails on maguro, but not on mako. Please either fix the test to be stable or disable it. No further releases will be allowed until this bug is fixed.

FAIL: ubuntuuitoolkit.tests.gallery.test_textinput.TextInputTests.test_textfield_numbers
File "/home/phablet/autopilot/ubuntuuitoolkit/tests/gallery/test_textinput.py", line 61, in test_textfield_numbers
self.assertEqual(textfield_numbers.text, '')

Update 20140129: bug #1267550 got fixed, but the issue is still showing up: http://ci.ubuntu.com/smokeng/trusty/touch/maguro/149:20140127:20140115.1/6305/ubuntu-ui-toolkit-autopilot/

[Cris Dywan (kalikiana)]

The problem is right there hidden in plain sight: the string can't be directly compared because that is racy. Instead Equally needs to be used as we do elsewhere. The reason it could work at all is probably python interpreter overhead.

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:ubuntu-ui-toolkit at revision None, scheduled for release in ubuntu-ui-toolkit, milestone Unknown

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-ui-toolkit - 0.1.46+14.04.20131216-0ubuntu1

---------------
ubuntu-ui-toolkit (0.1.46+14.04.20131216-0ubuntu1) trusty; urgency=low

  [ Zsombor Egri ]
  * Layouting fails with Qt5.2 if the tests and components are not
    rendered upon testing. Layout unit tests have been transferred to
    X11 unit tests and tests on layout changes all wait till the layout
    change completes. Tested with Qt 5.0.2 and Qt 5.2 RC1. (LP:
    #1256235)
  * Picker visuals facelift, prepared for Date- and TimePicker
    components.
  * Picker selectedIndex miss-behavior fix when model is
    cleared/cropped/reset.

  [ Florian Boucault ]
  * List item progression: use appropriate icon (same as Tabs' arrow)
    and fixes the issue of it being the wrong color in Dark and Gradient
    themes. (LP: #1236777)

  [ Albert Astals ]
  * tst_arguments: Do not use QTEST_MAIN QTEST_MAIN is 99.99% of the
    times what you want when running a Qt testcase but it creates a
    Q*Application and in this case that is not wanted since we are
    creating and destroying them as part of the test and Qt doesn't like
    when there's more than one Q*Application at the same time, so do
    without creating a Q*Application in main. This can of course have
    some implications but it seems that for this testcase all is fine.
    (LP: #1254747)
  * Make the model a var and not a ListModel ListModel is bound
    specifically to QQuickListModel By using a var we can use other
    QAbstractItemModel subclasses in the model.

  [ Nic ]
  * Removed stop animation on text every time an expansion/collapse is
    made. Fixes bug #1231939. (LP: #1231939)
  * Fixed scrolling in selector when parent widget's height is less than
    the selector's ListView height. Fixes bug #1256356. (LP: #1256356)

  [ Timo Jyrinki ]
  * Require either Qt 5.2 or libqt5v8-5-private-dev.

  [ tpeeters ]
  * Reset hide timer when user interacts with toolbar or toolbar
    buttons. (LP: #1249031)
  * Improve detection of tab bar interaction. Now using the new
    TabBar.pressed property instead of TabBar.selectionMode property,
    because selectionMode may be changed without user interaction (for
    example, when resuming an app in a following MR), and then the
    toolbar should not automatically hide.
  * Open the toolbar and tabbar (without animating) when the app is
    resumed. (LP: #1246790)

  [ Christian Dywan ]
  * Accept qmldir files in qmlapicheck and skip internal components.
    (LP: #1256045)
  * Suppress qdoc warnings coming from upstream API.
  * Conditionally define doc folder containing resources. (LP: #1256779)
  * Move doc commands to shellscript and add success output lines.
  * Use Eventually when checking that the textfield is empty There's
    always a delay so we can't just compare it. (LP: #1259476)

  [ Ubuntu daily release ]
  * Automatic snapshot from revision 892
 -- Ubuntu daily release <email address hidden> Mon, 16 Dec 2013 08:17:31 +0000

[Timo Jyrinki (timo-jyrinki)]

Reopening, because it continues to fail on maguro most of the time:

http://ci.ubuntu.com/smokeng/trusty/touch/maguro/114:20140106:20131223.2/5905/ubuntu-ui-toolkit-autopilot/

If it's now a different bug just affecting the same test, I can also keep this one fixed and open a new bug.

[Tim Peeters (tpeeters)]

I couldn't reproduce the bug in maguro image 79 with: for i in {0..100}; do autopilot run ubuntuuitoolkit.tests.gallery.test_textinput.TextInputTests.test_textfield_numbers; done

I'll update my image and try again.

[Tim Peeters (tpeeters)]

I couldn't reproduce with surfaceflinger on image 79 or 118 with maguro.

With image 118 on maguro with mir, I got this: https://pastebin.canonical.com/102651/ but at the same time I got disconnected from the device (via adb), the test-loop terminated, and the screen blanked so I am not sure what caused it.

Running:
for i in {0..100}; do autopilot run ubuntuuitoolkit.tests.gallery.test_textinput.TextInputTests.test_textfield_numbers; done
again gave me no failures.

One thing I notice that seems wrong, is while the test is running and numbers are being typed, the OSK is showing letters, not numbers.

[Tim Peeters (tpeeters)]

I started webbrowser by tapping its icon, and then ran the text_textfield_numbers test twice, and I reproduced the error

[Cris Dywan (kalikiana)]

I can see the exact failure (only) if I first have any other app running before starting the test runs like Tim above. Consistently the second iteration fails.

[Tim Peeters (tpeeters)]

so, with webbrowser open, the first time I run the test is always fine.

Then the test finishes and I'm returned to the webbrowser, but the keyboard is still opened. Running hte test again (2nd time), the uitk-gallery is opened and the textfield page is selected, and the textfield itself becomes activated for a few seconds, but its contents is not edited. Then there is a few seconds delay, and the test fails and is terminated.

Executing the test a third time gives the same results as the first time (it passes, but keyboard stays open, and the fourth time it fails again, etc).

[Tim Peeters (tpeeters)]

note: after the failing test, I'm returned to the webbrowser app, without the OSK open. So then everything works fine again (until after I run the test again).

[Tim Peeters (tpeeters)]

I get exactly the same behavior (when webbrowser is open, test fails on second run) with ubuntuuitoolkit.tests.gallery.test_textinput.TextInputTests.test_textfield_standard so it is not related to the specific numbers test. Perhaps there is a problem with OSK focus sending input to the wrong app? And the OSK is not properly closed after the first test run.

[Bill Filler (bfiller)]

This in fact is caused by bug https://bugs.launchpad.net/ubuntu/+source/unity-mir/+bug/1267550
The osk is not being properly closed when the app is quit and another app is running. Closing this bug and will track in new bug.

[Timo Jyrinki (timo-jyrinki)]

Reopening (de-duplicating) since bug #1267550 got fixed, but the issue is still showing up: http://ci.ubuntu.com/smokeng/trusty/touch/maguro/149:20140127:20140115.1/6305/ubuntu-ui-toolkit-autopilot/

[Timo Jyrinki (timo-jyrinki)]

Putting also to UI Toolkit now again, since if there's no other solution the test needs to be disabled before the next release.

[Cris Dywan (kalikiana)]

We meanwhile got new emulators which may make the test more robust, I'll have a branch for using them soon. I'll also be checking if I can reproduce with the latest image.

[Tim Peeters (tpeeters)]

Can the issue in this MR https://code.launchpad.net/~zsombi/ubuntu-ui-toolkit/dateUtilsTestFix/+merge/203700 also be caused by the bug here? It seems like one typed letter did not end up in the correct textfield.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/ubuntuuitoolkit/tests/test_emulators.py", line 920, in test_write
    self.assertEqual(self.simple_text_field.text, 'test')
  File "/usr/lib/python2.7/dist-packages/testtools/testcase.py", line 322, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/usr/lib/python2.7/dist-packages/testtools/testcase.py", line 412, in assertThat
    raise MismatchError(matchee, matcher, mismatch, verbose)
MismatchError: u'est' != 'test'

[Cris Dywan (kalikiana)]

The first time I see it fail in that location. If this is the same losing focus/osk it could well be it, symptoms-wise.

If it can fail there disabling tests is not desirable, the emulators which are used by apps would merely fail there in much more confusing ways than here.

[Cris Dywan (kalikiana)]

I cannot manage to reproduce on my device except by locking the screen. Has anybody else tried on their device?

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:ubuntu-ui-toolkit at revision 932, scheduled for release in ubuntu-ui-toolkit, milestone february-2014

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-ui-toolkit - 0.1.46+14.04.20140212-0ubuntu1

---------------
ubuntu-ui-toolkit (0.1.46+14.04.20140212-0ubuntu1) trusty; urgency=low

  [ Andrew Hayzen ]
  * Fixed text not aligned at vertical centre if the listitem height has
    changed dynamically. Fixed no removeItemAnimation if the listitem
    height has been set. (LP: #1263688, #1263682)

  [ CI bot ]
  * Trunk as is, for CI Train release.

  [ William Hua ]
  * Also update LANGUAGE when setLanguage is called for dynamic
    translation. (LP: #1263163). (LP: #1263163)

  [ Timo Jyrinki ]
  * A qtdeclarative5-private-dev packaging change adding a dependency on
    libqt5v8-5-private-dev has been backported to all Ubuntu versions
    now, so this workaround can now be dropped - building both against
    Qt 5.2 and 5.0.2 works.
  * Add #include for QDebug to fix Qt 5.2 build.

  [ Christian Dywan ]
  * Split build and install steps in debian/rules.
  * Use integer Date constructor in date picker tests for Qt 5.2.
  * Run unit_x11 test cases via Xvfb and address failures. (LP:
    #1258017, #1237812, #1242646)
  * Don't redirect output of the input file in itself.
  * Don't try to deploy non-existing qml and js files. (LP: #1259228)
  * Assert textField focus, button visibility and pressed after tapping
    clear.
  * Split documentation for previously group properties. (LP: #1266842)
  * Suppress errors to work-around font error messages. (LP: #1256999)
  * ListItems.Empty's default property needs to be data not children.
    (LP: #1190509)

  [ Pete Woods ]
  * Handle the unity-action-api quit signal. (LP: #1269409)

  [ Zsombor Egri ]
  * Date picker component. Provides full date and month only picking
    functionality.
  * DatePicker test case fix, setting the minimum limit to value that is
    earlier than the date value set. (LP: #1266515)
  * Fixing state restoration when nested dynamic component properties
    are restored. (LP: #1267039)
  * Removing obsolete functionality from QuickUtils. (LP: #1266707)
  * StateSaver failure with Repeater fixed. Indirectly fixes StateSaver
    with ListView and GridView. (LP: #1267900)
  * TabBar test case failure fix for Qt5.2. (LP: #1256930)
  * InverseMouseArea API test removed, being a duplicate of the other
    InverseMouseArea tests. Connecting to th eproper enabledChanged
    signal in InverseMouseArea resolves the warning in Qt5.2 stating
    enabledChanged was overwritten in QQuickMouseArea. (LP: #1266707)
  * Panel implementation for DatePicker. Presents the DatePicker either
    in a panel covering the OSK area or in a Popover, depending on the
    form factor and presence of input method provider.
  * Layout tests now wait for the currentLayout change completion. (LP:
    #1266707)
  * Extra QQmlEngine instance removed from plugin, which seemed to cause
    some race conditions with Qt5.2. As result UbuntuColors got
    transferred from context property into singleton object, Theme's
    palette creation delayed till its first access. (LP: #1266707)
  * DateUtils test failure fix. (LP: #1273893)

  [ Florian Boucault ]
  * Ubuntu UI Toolkit Gallery: reimplemented in a cleaner way the
    respo...