xserver mouse pointer emulation from touch breaks with QML app.

Bug #1307701 reported by Steve Magoun on 2014-04-14
40
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OEM Priority Project
Critical
Unassigned
Trusty
Critical
Unassigned
X.Org X server
Confirmed
Medium
qtbase-opensource-src (Ubuntu)
High
Maarten Lankhorst
Trusty
High
Timo Jyrinki
unity (Ubuntu)
High
Daniel d'Andrada
Trusty
Undecided
Unassigned
xorg-server (Ubuntu)
High
Maarten Lankhorst
Trusty
Undecided
Unassigned

Bug Description

When running an SDK app on the desktop, it's not possible to use the touchscreen to move windows, use the launcher, etc. The SDK app seems to consume all touchscreen events.

To reproduce:
1) On a desktop/laptop with a touchscreen (I am using a Dell XPS13), install 14.04 and check that the touchscreen works for moving windows
2) Install the camera-app package from universe (I have 2.9.1+14.04.20140331-0ubuntu1)
3) Launch the camera app
4) Touch once inside the camera-app window
5) Use the touchscreen to drag the camera-app window

Expected results:
You can move the camera-app window using the touchscreen

Actual results:
You cannot move the camera-app window, or perform any other touch tasks in unity, with the touchscreen

This only affects touchscreen input - mouse/touchpad input is not affected

This affects all apps written with the SDK (I tried camera-app, gallery-app, reminders-app). It's 100% reproducible.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: unity 7.2.0+14.04.20140411-0ubuntu1
ProcVersionSignature: Ubuntu 3.13.0-24.46-generic 3.13.9
Uname: Linux 3.13.0-24-generic x86_64
ApportVersion: 2.14.1-0ubuntu2
Architecture: amd64
CompizPlugins: [core,commands,composite,opengl,compiztoolbox,decor,vpswitch,snap,mousepoll,resize,place,move,wall,grid,regex,imgpng,session,gnomecompat,animation,fade,unitymtgrabhandles,workarounds,scale,expo,ezoom,unityshell]
CurrentDesktop: Unity
Date: Mon Apr 14 14:59:09 2014
DistributionChannelDescriptor:
 # This is a distribution channel descriptor
 # For more information see http://wiki.ubuntu.com/DistributionChannelDescriptor
 canonical-oem-somerville-precise-amd64-20130203-1
EcryptfsInUse: Yes
InstallationDate: Installed on 2013-12-02 (132 days ago)
InstallationMedia: Ubuntu 12.04 "Precise" - Build amd64 LIVE Binary 20130203-13:50
SourcePackage: unity
UpgradeStatus: Upgraded to trusty on 2014-02-12 (61 days ago)

Steve Magoun (smagoun) wrote :
tags: added: rls-t-incoming
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in unity (Ubuntu):
status: New → Confirmed
tags: added: desktop-touch
removed: rls-t-incoming
Olli Ries (ories) wrote :

can you reproduce on another touch screen? It seems to work fine on my XPS12

Olli Ries (ories) on 2014-04-15
Changed in unity (Ubuntu):
assignee: nobody → Stephen M. Webb (bregma)
importance: Undecided → High
Olli Ries (ories) wrote :

happens on the XPS12 too once you click _into_ an app

some more observations:
- once the bug is activated (by clicking into the touch app), all touch screen events seem to be either discarded (when not in the app surface) or consumed (when in app surface)
- closing the app releases the events and the touch screen works fine again

Olli Ries (ories) wrote :

also, happens with a non-SDK app, e.g. from http://qt-project.org/doc/qt-5/qml-tutorial2.html

import QtQuick 2.0

Item {
    id: container
    property alias cellColor: rectangle.color
    signal clicked(color cellColor)

    width: 40; height: 25

    Rectangle {
        id: rectangle
        border.color: "white"
        anchors.fill: parent
    }

    MouseArea {
        anchors.fill: parent
        onClicked: container.clicked(container.cellColor)
    }
}

Ara Pulido (ara) on 2014-04-23
Changed in oem-priority:
importance: Undecided → Critical
Stephen M. Webb (bregma) on 2014-04-23
Changed in unity (Ubuntu):
status: Confirmed → In Progress
tags: added: qa-touch
summary: - Unity does not get touch events when SDK apps running
+ Unity does not get touch events when QML apps running

On a side note: multitouch in xorg is very problematic. Just now I had to reboot my xps touch screen laptop because the mouse pointer emulation logic no longer worked (pointer simply disappeared on touch). In another attempt playing with multitouch, xserver simply crashed and I was sent back to the login screen.

Daniel d'Andrada (dandrader) wrote :

So, when you run a QML app and tap on it, it seems that the mouse pointer emulation dies but multitouch events still flow normally as unity still responds to 3-finger (window drags, "alt-tab" switch) and 4-finger (show/hide dash, launcher drag) gestures

Daniel d'Andrada (dandrader) wrote :

hmm, and pointer emulation actually still works for other apps when their windows are focused. You just can't focus them by tapping them.

Stephen M. Webb (bregma) wrote :

The problem does not occur in a Gnome Flashback (Compiz) session, so it is evidently a result of the interaction between the XInput2 touch grabs used in Unity 7 (through Geis), the passive Xinput touch subscriptions in Qt, and the pointer emulation in x.org.

Multi-touch pointer emulation in XInput2 is documented as acting the way as described in this bug when multi-touch sequences are neither accepted nor rejected (ie pointer emulation stops when the first touch ends - see http://who-t.blogspot.ca/2011/12/multitouch-in-x-pointer-emulation.html) although it is unclear why the emulation does not resume when all touches have stopped.

It sounds like the Unity touch support needs to add gesture accept/reject.

Stephen M. Webb (bregma) wrote :
Download full text (3.3 KiB)

Here is a test setup:

(1) install ubuntu-clock app
(2) modify /usr/share/gnome-session/sessions/ubuntu.session to remove the 'compiz;' entry so compiz gets started by upstart instead of gnome-session
(3) modify /usr/share/upstart/session/unity7.conf to add the following lines in the pre-start script stanza:
  initctl set-env --global GEIS_DEBUG=3
  initctl set-env --global GRAIL_DEBUG=-1
(4) log in to Unity, start the clock app and touch it anywhere in the window

The .cache/upstart/unity7.log contains the following.

[CLOCK] ClockPage loaded
[CLOCK] Saved Current Location: Merrickville
[CLOCK] Saved Current Location Coordinates: -75.8333,44.9167
[CLOCK] Alarm Database loaded
[CLOCK] MainView loaded
[CLOCK] Retrieving Sunrise/Sunset times from disk
GEIS(debug)-geis_backend_multiplexor_pump:414 activity 0x1 on fd 18 callback_info=0x1e62f50
GEIS(debug)-geis_backend_multiplexor_pump:414 activity 0x1 on fd 19 callback_info=0x1e655e0
GRAIL DEBUG (regular-recognizer.cpp:ProcessFrameEvent:54): new event 0x4997c80 with time 161447105
GRAIL DEBUG (regular-recognizer.cpp:ProcessTouches:95): touch 1 began with start time 161447105
GRAIL DEBUG (regular-recognizer.cpp:ProcessTouches:98): touch 1 has been added to all_touches_
GRAIL DEBUG (regular-recognizer.cpp:ProcessTouches:99): touch 1 has been added to free_touches_
GRAIL DEBUG (regular-recognizer.cpp:ProcessFrameEvent:54): new event 0x42edc80 with time 161447108
GRAIL DEBUG (touch.cpp:Update:65): touch 1 is owned
GEIS(debug)-geis_backend_multiplexor_pump:414 activity 0x1 on fd 18 callback_info=0x1e62f50
GRAIL DEBUG (handle.cpp:UpdateTime:197): client updating time to 161447165
GRAIL DEBUG (recognizer.cpp:UpdateTime:304): Updating time to 161447165
GRAIL DEBUG (recognizer.cpp:RejectOverdueGesturesAndTouches:288): touch 1 has been removed from free_touches_ because it is older than the gesture composition time (time: 161447165, touch start time: 161447105)
GRAIL DEBUG (recognizer.cpp:RejectOverdueGesturesAndTouches:292): touch 1 has been erased from free_touches_
GRAIL DEBUG (touch.cpp:~Touch:80): rejecting touch 1
GRAIL DEBUG (recognizer.cpp:UpdateTime:304): Updating time to 161447165
GEIS(warning)-_x11_fd_callback:1160 failed to get X generic event data
GEIS(warning)-_x11_fd_callback:1160 failed to get X generic event data
GEIS(debug)-geis_backend_multiplexor_pump:414 activity 0x1 on fd 19 callback_info=0x1e655e0
GRAIL DEBUG (regular-recognizer.cpp:ProcessFrameEvent:54): new event 0x42edc80 with time 161447166
GRAIL DEBUG (regular-recognizer.cpp:ProcessTouches:126): touch 1 has been erased from all_touches_

From this log you can see that the single touch is reported to Unity and gets rejected because additional touches do not begin within the gesture setup period.

According to the XInput2 documentation, this should forward the TouchBegin to the clock, which should then do with it what it will. The clock should not be receiving emulated pointer events, because according to the XInput2 documentation, if Touch events are explicitly selected, pointer emulation is disabled for that client.

It looks like either (a) the X server is in pointer emulation mode and never leaves, or (b) the client is doing ...

Read more...

Changed in unity (Ubuntu):
assignee: Stephen M. Webb (bregma) → Daniel d'Andrada (dandrader)
Daniel d'Andrada (dandrader) wrote :

Any QML application at all will do (such as "Item {width:600;height:600}")

What seems to be happening, from what I've learned so far:

- user touches on a qml application
- xserver sends the resulting touch events to both the qmlscene and to compiz/unity7
- xserver offers ownership to compiz/unity7
- qmlscene does XIGrabDevice on the touch screen
- compiz/unity7 rejects the touch
- user lifts finger and touch ends
- qmlscene does XIUngrabDevice on the touch screen
- user lands a new finger, this time on the desktop
- xserver still has in its bookkeeping that the previous touch is active and thus deny pointer emulation for that new touch

And from that point onwards xserver is left with a bogus state, even after the qml app is closed. Restarting it being the sure way to clean it up and reproduce the issue anew (otherwise further issues seem to crop up)

Alberto Milone (albertomilone) wrote :

@Daniel, have you tried adding some qDebug() lines in the QT5 libraries to see if we actually get the touch events and finally ungrab the device? (the code is in src/plugins/platforms/xcb/qxcbconnection_xi2.cpp)

Also, it would be interesting to see what XIGrabDevice and XIUngrabDevice return.

Maarten Lankhorst (mlankhorst) wrote :

dandrader: That seems useful for making a testcase, thanks!

Daniel d'Andrada (dandrader) wrote :

One thing that smells fishy is that the qml app starts getting touch events even before compiz/unity7. I think that this should not happen as the qml app does not subscribe for ownership events (unlike compiz/unity7) and ownership is offered to compiz/unity7.

So the qml app should only get touch events once compiz/unity7 rejects ownership of the touch sequence, but that's just not happening (qml app was already receiving events from that touch sequence by that time).

Alberto Milone (albertomilone) wrote :

On my system the touchscreen keeps working after using a QML app. It still gets gestures (e.g 3 fingers on the screen) but it won't work on some windows. Furthermore, I can't drag windows by the decoration any more.

Daniel d'Andrada (dandrader) wrote :

> On my system the touchscreen keeps working after using a QML app. It still gets gestures (e.g 3 fingers on the screen) but it won't work on some windows. Furthermore, I can't drag windows by the decoration any more.

Multifinger gestures keep working because they do not depend on pointer/mouse events, which is what gets broken here (namely "mouse pointer emulation from touch events"). They use touch events directly.

summary: - Unity does not get touch events when QML apps running
+ xserver mouse pointer emulation from touch breaks with QML app.
Daniel d'Andrada (dandrader) wrote :

Took the liberty of correcting the bug title as unity does keep getting touch events. It's the xserver mouse pointer emulation from touch events that breaks badly.

Daniel d'Andrada (dandrader) wrote :

Attaching detailed xserver log. The following steps were taken:

1- sudo restart lightdm (to restart the X server. so that de have a clean slate)
2- run a very simple qml app with qmlscene
3- tapped on that qmlscene window (touch id 1)
4- tapped on the desktop area (nautilus) (touch id 2)
5- sudo stop lightdm

From the logs you can see that no pointer emulation was done for touch id 2 because xserver still thinks that touch id 1 is active.

Daniel d'Andrada (dandrader) wrote :

xserver packages I used to take the logs above.

Daniel d'Andrada (dandrader) wrote :

Excerpt from #xorg-devel:

"""
<dandrader> daniels, I've the following situation: clientX has a XISelectEvents(TouchBegin|Update|End, childWindowA, slaveDevice1); clientY has a XIPassiveGrabDevice(TouchBegin|Update|End|Ownership, rootWindow, AllMasterDevices)
 daniels, when a touch reaches xserver, it sends touch events to both clientX and clientY
<daniels> dandrader: yes, events are delivered separately thorough master and slaves - i'd recommend not using slaves for anything at all
<dandrader> daniels, but I believe it should send only to clientY, and only once clientY rejects the touch sequence should clientX finally start receiving the touch events (replayed), right?
<daniels> dandrader: if both clients were listening on a master device, then you'd be right
<dandrader> daniels, hmm got it. so what I'm seeing is the expected behavior then and I should modify clientX to select events on AllMasterDevices instead of on slaveDevice1
 daniels, is that it?
<daniels> dandrader: exactly
"""

So we have two problems here:
  1- touch ownership mechanics won't work in our situation as Qt is talking directly to the slave devices whereas unity7 is talking to the master devices.
   2- the xserver bug described in my previous comments where it gets confused with such situation and ends up with a borked internal state which ruins mouse pointer emulation

So if we could avoid the situation in item 1 (by changing either Qt or unity7/"Open Input Fw Grail" code) we might (have to try it out to see) circumvent the bug in item 2.

Alberto Milone (albertomilone) wrote :

Is there a specific reason for calling XISelectEvents() for each slave vs using XIAllMasterDevices in QXcbConnection::xi2Select() in Qt?

Daniel d'Andrada (dandrader) wrote :

 > Is there a specific reason for calling XISelectEvents() for each slave vs using XIAllMasterDevices in QXcbConnection::xi2Select() in Qt?

Would have to dig into Qt code to tell that. But my guess is that because it exposes QTouchDevice objects to applications. For every QTouchEvent you can get the QTouchDevice from where it came from. But it might be that it could work with XIAllMasterDevices just as well.

Stephen M. Webb (bregma) wrote :

The code in Qt that subscribes to slave device events is the explicit tablet-handling code. Checking the code, the touchscreen on my Dell XT2 is being treated as a tablet by the Qt code because it supports a stylus mode on one of its alternate devices.

It sounds like the bug is in the Qt XCB QPA code, which explains why it only started occuring in QML apps after the switch to Qt 5.2.

Maarten Lankhorst (mlankhorst) wrote :

I think the XIDeviceEvent.sourceid would tell you from which device it originates. The XIDeviceEvent.deviceid member would be set to the master device, which might break things in a straight conversion.

Alberto Milone (albertomilone) wrote :

@Stephen: I guess that means that your device passes the following check:

if (tabletData.valuatorInfo.contains(QXcbAtom::AbsX) &&
     tabletData.valuatorInfo.contains(QXcbAtom::AbsY) &&
     tabletData.valuatorInfo.contains(QXcbAtom::AbsPressure))

which you can check using xinput list-props $device_id or simply using evtest.

I have a non-tablet and a tablet, they are both affected by the same problem, except that fixing the tablet use case will probably take a little more work.

Alberto Milone (albertomilone) wrote :

I've just rebuilt Qt with a partial fix which only covers the "non-tablet" test case, and I'm glad to report that it solves the problem.

Maarten is working on an all comprehensive fix.

It happens on Ubuntu running on laptop with a touchscreen.

Scenario:

- There's a small window whose client called XISelectEvents on it for the touchscreen slave device with mask=XI_TouchBegin|XI_TouchUpdate|XI_TouchEnd (client[5290, /usr/lib/x86_64-linux-gnu/qt5/bin/qmlscene qml-demo-minimal-client.qml] in the logs).

- compiz and nautilus (which handles the desktop background) have selections and/or grabs on the master device

Actions:

1- user taps the qmlscene window. Touch events are reported to both qmlscene (from the slave device) and to compiz (from the master device). compiz rejects the touch sequence and it's not delivered to anyone else

2- user now presses on the desktop background. TouchBegin, TouchOwnership and further TouchUpdates are sent to compiz.

3- compiz rejects the touch sequence. TouchPuntToNextOwner fails to emulate a pointer and pass it to nautilus as it still thinks that touch 1 is active. And as per pointer emulation rules, only the oldest active touch is used for touch emulation. From the logs, the lines are:

"""
[ 9223.917] (II) TouchPuntToNextOwner: touch 2
[ 9223.917] (II) - Not all older pointer emulated touches have been seen yet. Oldest touch(client_id=1, active=1, pending_finish=0, emulate_pointer=1).
"""

4- user lifts his finger

So that scenario on touch 1 clearly left xserver in an inconsistent, buggy, state.

Created attachment 98564
Xorg.log with detailed input event handling info

Created attachment 98565
A filtered Xorg.log with only the interesting parts

Created attachment 98566
output of xinput

Created attachment 98567
output of xwininfo showing the whole window tree

For reference, those were the steps I took when recording the log

- started lightdm (and threfore X)
- logged in
- launched the Qt app (qmlscene)
- tapped on the qmlscene window
- tapped on the desktop background (nautilus)
- closed the Qt app (qmlscene)
- stopped lightdm (and threfore X)

Daniel d'Andrada (dandrader) wrote :

And here's the upstream bug I reported:
https://bugs.freedesktop.org/show_bug.cgi?id=78345

Alberto Milone (albertomilone) wrote :

@Daniel: thanks for taking the time to file an upstream bug report.

I have tested Maarten's latest work on the Qt library, and I can confirm that it solves the issue on both tablets and non-tablets. Maarten's work is available here (you only need to update the qt packages):
https://launchpad.net/~canonical-x/+archive/x-staging

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in qtbase-opensource-src (Ubuntu):
status: New → Confirmed
affects: xorg-server (Ubuntu) → qtbase-opensource-src (Ubuntu)
Changed in qtbase-opensource-src (Ubuntu):
assignee: nobody → Maarten Lankhorst (mlankhorst)
importance: Undecided → High
status: New → In Progress
Daniel d'Andrada (dandrader) wrote :

I confirm that the Qt from https://launchpad.net/~canonical-x/+archive/x-staging does XISelectEvents on the master pointer (instead of the touchscreen slave device, as previously) and that the resulting behavior of the xserver is good. Ie., touch ownership mechanics work properly and mouse pointer emluation from touch events keeps working as well.

Daniel d'Andrada (dandrader) wrote :

Qt still XISelectEvents my "N-Trig Pen stylus" and "N-Trig Pen eraser" slave devices though. But thankfully no events come from them and thus peace is maintained :)

Changed in unity (Ubuntu):
status: In Progress → Invalid

Created attachment 98629
xorg-integration-tests testcase

I think this reproduces the bug quite well, does this match your logs?

Changed in xorg-server (Ubuntu):
assignee: nobody → Maarten Lankhorst (mlankhorst)
importance: Undecided → High
status: New → In Progress

Comment on attachment 98629
xorg-integration-tests testcase

Woops, test is broken, the way it was written probably doesn't reproduce this bug. Although I believe it probably matches the description of the bug. Maybe missing something?

Changed in xorg-server:
importance: Unknown → Medium
status: Unknown → Confirmed

Created attachment 98694
xorg-integration-tests testcase try 2

Ok so I've finally isolated the problem. The problem is the XIGrabDevice on the slave pointer, which is only removed after the touch ends. When this happens the ButtonRelease call for pointer emulation is never generated, leading to the 'stuck button' bug.

Modifying qt to listen to touch events on the master fixes it and removes the need for the XIGrabDevice call. The grab was originally done to prevent pointer emulation events, but these won't get generated when we listen for touch events on the master, solving both problems.

But this is still a bug in xorg-server, regardless. :)

Changed in qtbase-opensource-src (Ubuntu Trusty):
status: New → In Progress
importance: Undecided → High
assignee: nobody → Timo Jyrinki (timo-jyrinki)
Changed in unity (Ubuntu Trusty):
status: New → Invalid
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qtbase-opensource-src - 5.2.1+dfsg-1ubuntu17

---------------
qtbase-opensource-src (5.2.1+dfsg-1ubuntu17) utopic; urgency=medium

  * Disable PCH also on arm64 in addition to armel/armhf (LP: #1318635)
 -- Timo Jyrinki <email address hidden> Tue, 13 May 2014 04:37:13 +0000

Changed in qtbase-opensource-src (Ubuntu):
status: In Progress → Fix Released
Timo Jyrinki (timo-jyrinki) wrote :

qtbase 5.2.1+dfsg-1ubuntu14.2 SRU for trusty is in unapproved queue now too (https://launchpad.net/ubuntu/trusty/+queue?queue_state=1&queue_text=)

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in xorg-server (Ubuntu Trusty):
status: New → Confirmed

Hello Steve, or anyone else affected,

Accepted qtbase-opensource-src into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/qtbase-opensource-src/5.2.1+dfsg-1ubuntu14.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in qtbase-opensource-src (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: added: verification-needed
Steve Magoun (smagoun) wrote :

I can no longer reproduce this with qtbase 5.2.1+dfsg-1ubuntu14.2 - nice work!

tags: added: verification-done
removed: verification-needed
Ara Pulido (ara) on 2014-05-29
Changed in oem-priority:
status: New → Fix Committed

The verification of the Stable Release Update for qtbase-opensource-src has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qtbase-opensource-src - 5.2.1+dfsg-1ubuntu14.2

---------------
qtbase-opensource-src (5.2.1+dfsg-1ubuntu14.2) trusty; urgency=medium

  * debian/patches/xi2-use-master-device.patch:
    - Make xi2 select events on master device instead of slaves.
      (LP: #1307701)

qtbase-opensource-src (5.2.1+dfsg-1ubuntu14.1) trusty; urgency=medium

  [ Dmitry Shachnev ]
  * Backport upstream patch to fix issues with keymap update handling
    (Add_better_support_for_keymap_update_handling.patch, copied from
    Debian and rebased). (LP: #1318482)
  * Build-depend on libxkbcommon-x11-dev, as the new patch includes
    <xkbcommon/xkbcommon-x11.h>.
 -- Maarten Lankhorst <email address hidden> Tue, 06 May 2014 13:14:17 +0000

Changed in qtbase-opensource-src (Ubuntu Trusty):
status: Fix Committed → Fix Released
Ara Pulido (ara) on 2014-06-10
Changed in oem-priority:
status: Fix Committed → Fix Released
CSRedRat (csredrat) wrote :
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.