Sometimes input breaks and only edges are responsive

[Oliver Grawert]

I had the music app (paused) open over night on my mako, waking up the device I found the application page showing me the one open app. The shell page itself is completely unresponsive, all side edge swipes seem to work fine though. I can switch to the music app when swiping from the right, bring up the launcher from the left and use the indicators from the top.

When I am in the music app I can bring up the HUD from the bottom (trying to close the app, since the closing via the unresponsive shell does not work) but the HUD is completely unresponsive. Trying to close the HUD again via swiping down I see the indicators expand behind it and I am able to get input into the expanded indicators behind the HUD, but not the HUD itself.

TEST CASE:
1. tap and hold the dash (do not release)
2. with the other hand, drag in the launcher and launch an app
3. when the other app has started, release the first finger

ACTUAL RESULT
Dash mouse input will be broken. You cannot scroll the dash any more or tap entries. However, touch input still works as you can still activate the bottom edge.

EXPECTED RESULT
Dash is still responding to mouse input and touch input.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: unity8 7.84+14.04.20140319.1-0ubuntu1
Uname: Linux 3.4.0-5-mako armv7l
ApportVersion: 2.13.3-0ubuntu1
Architecture: armhf
Date: Fri Mar 21 11:57:53 2014
InstallationDate: Installed on 2014-03-20 (1 days ago)
InstallationMedia: Ubuntu Trusty Tahr (development branch) - armhf (20140320)
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
SourcePackage: unity8
UpgradeStatus: No upgrade log present (probably fresh install)

[Michał Sawicz (saviq)]

This is most probably about mouse (as opposed to touch) events getting lost, edges use touch events while all the rest uses mouse events, and those seem to get lost some times.

I saw that once or twice without wake/resume, so that doesn't seem to be important.

[Michał Sawicz (saviq)]

This seems to happen more with new scopes, still a case of "edges work, no other input does"...

[Pat McGowan (pat-mcgowan)]

Upon loading build 258 this same symptom manifested itself a couple of times.

First boot - swipe from apps to music to video. The UI is very sluggish and then it freezes half way through the transition to the video scope. The system turned the screen off, power button to bring it back went to the apps page, which was still hung. A few moments later it started to work again.

Second occurrence a moment later - opened the gallery, returned to the dash apps page, and UI hung.
Returning to the gallery with a right edge swipe and then back to the dash cleared the hand.

Looking at top there was nothing apparent during the UI hangs.

Subsequently I am unable to reproduce it. The device has around 20 mp3, 7 videos, and 50 photos.

[Omer Akram (om26er)]

I have been trying to find out exact steps for this bug for last hour but have not been able to find a pattern. Though I did indeed start seeing it with new-scopes very often.

[Michał Sawicz (saviq)]

This became less frequent, but still a big issue. Raising importance because we really need to get to the heart of this.

[Daniel d'Andrada (dandrader)]

I need steps to reproduce it.
Never experienced this when playing with the device.

[Michał Sawicz (saviq)]

Yeah problem is there are no clear steps, this "just happens" from time to time. I have a feeling this is related to the device suspending, so don't keep it connected to usb.

If all else fails, we might need to resort to remote access to a device someone got locked up.

[Michał Sawicz (saviq)]

Has anyone ideas about reproducing this?

[Jean-Baptiste Lallement (jibel)]

There is a way to reproduce specific to ubuntu-system-settings, the symptoms are similar but I am not sure the cause is the same.

Test Case:
1. Launch system settings
2. Tap on accounts
3. Tap on back
4. There is a black screen for 20s (bug 1292652). Wait until it goes away and u-s-s is displayed

-> At this point launcher, indicators and dash are working (edges are responsive) but u-s-s doesn't receive input event (cannot scroll the screen or tap an icon)

[Michał Sawicz (saviq)]

I'm afraid the above is different - bug #1355263

[Daniel d'Andrada (dandrader)]

I wonder if this could be caused by a leftover MouseArea{} event eater, like the one used by notifications...

[Michał Sawicz (saviq)]

That is an interesting idea...

[Daniel d'Andrada (dandrader)]

So the latest theory is that mouse pointer emulation out of touch events inside Qt (more specifically, inside QQuickWindow) breaks for some reason. So all MouseArea elements in unity8 stop getting events but other elements that are touch-event based like the indicator panel and edge gestures keep working.

[Daniel d'Andrada (dandrader)]

Upstream fix (or work around, depending on how you see it) here:
https://codereview.qt-project.org/#/c/93002/

[Timo Jyrinki (timo-jyrinki)]

Upstream seems to currently think (as per given code reviews) it's not the right level to do a workaround at, ie. platform should behave correctly or constrain the problematic behavior at a lower level. Although, the debate was still ongoing 3 days ago whether it's a realistic scenario that one holds touch so still that it doesn't generate any events - if it's unlikely enough, a time limit approach could be used when no new touch updates are received for a certain time and a new touch begins. The discussion could probably continue with a new patch set with the time limit approach.

[Daniel d'Andrada (dandrader)]

@Timo

I decided to do the touch stream validation at qtmir level instead, ensuring qtmir always provides a sane and clean stream of touch events to Qt regardless of what comes to it, thus avoiding Qt from getting stuck at that bogus state. Or so I hope :)

[Daniel d'Andrada (dandrader)]

But of course it would be nice to have Qt more resilient/robust as well.

[Alexander Sack (asac)]

would be good to get the fix landed in rtm today or tomorrow so we can use a promoted image to hand to customer QA team this week..

[Michael Zanetti (mzanetti)]

Here's a way to easily reproduce it with the dash to verify if this issue is fixed:

* tap and hold the dash (do not release)
* with the other hand, drag in the launcher and launch an app
* when the other app has started, release the first finger

=> Dash mouse input will be broken. You cannot scroll the dash any more or tap entries. However, touch input still works as you can still activate the bottom edge.

[Daniel d'Andrada (dandrader)]

@Zanetti

> Here's a way to easily reproduce it with the dash to verify if this issue is fixed:
> [...]

Thanks for that test case! Very useful.

That's another issue, which lp:~dandrader/qtmir/missingTouchEnd-lp1295623 doesn't fix. I was hoping to deal with it in another bug report but we can keep using this one since to the user it's hard to distinguish between the two.

So, those are the problems:

1 - unity8 not receiving a touch end from Mir (or lower, like the evedev device) and thus having its mouse event emulation getting stuck
2 - A MirSurfaceItem inside unity8's QML scene not receiving a touch end, and thus leaving the mouse event emulation of the corresponding Qt client app stuck (like unity8-dash).

So lp:~dandrader/qtmir/missingTouchEnd-lp1295623 fixes item 1. Now I'm working on a fix/workaround for item 2. Item 2 does look like a Qt bug (although some might claim we're using its API wrongly). It happens when you disable an Item while it's owning a touch point, which causes QQuickWindow's event delivery to stop sending touch events for that Item (which is ok). But it will never get further updates for that touch point even when reenabled. Meaning it will never get a TouchEnd for that touch. When you tap on it again it receives a whole new TouchBegin like if the previous touch point, whose end is still pending, never existed. So, when you disable an Item that owns a touch point I think QQuickWindow should synthesize a TouchEnd for that disabled Item.