Multiple finger touch fails on Sony Vaio Duo 11 running Ubuntu 12.10 (N-Trig DuoSense MultiTouch)

[Konstantinos Togias]

N-trig DuoSense multitouch screen on Sony Vaio Duo 11 (model SVD1121P2EB) ultrabook/tablet hybrid does not function as expected with ubuntu 12.10.

How to reproduce the bug:

1. Boot up Sony Vaio Duo 11 to Ubuntu 12.10 and login to untiy
2. Use mouse/trackpad to Open a nautiuls window
3. Use a mouse/trackpad to Open a terminal window and run "tail -f /var/log/Xorg.0.log"
4. With the focus to the terminal window touch inside nautilus window with one finger. Nautilus window takes focus, as expected. Nothing is appended to Xorg.0.log.
5. Touch with one finger inside terminal window. The terminal window takes the focus, as expected. Nothing is appended to Xorg.0.log.
6. Touch with one finger inside nautilus window and do a drag gesture. A rectangle select area follows hand's move. Nothing is appended to Xorg.0.log.
7. Touch again with one finger inside nautilus window and and do a drag gesture. Nautilus window is scrolling up and down, as expected.
8. Touch with 2 fingers into nautilus window. The following Error line is appended to Xorg.0.log:

(EE) [dix] N-trig DuoSense: unable to find touch point 1

9. Do any gesture with the two fingers (eg. spread). The following error lines are repeated during the gesture:

[ 2530.712] (EE) BUG: triggered 'if (!(event->device_event.flags & (1 << 5)))'
[ 2530.712] (EE) BUG: ../../dix/touch.c:628 in TouchConvertToPointerEvent()
[ 2530.712] (EE) Non-emulating touch event
[ 2530.712] (EE)
[ 2530.713] (EE) Backtrace:
[ 2530.713] (EE) 0: /usr/bin/X (xorg_backtrace+0x36) [0x7f261cfadb76]
[ 2530.713] (EE) 1: /usr/bin/X (0x7f261ce05000+0x7dbdf) [0x7f261ce82bdf]
[ 2530.713] (EE) 2: /usr/bin/X (0x7f261ce05000+0x1380b9) [0x7f261cf3d0b9]
[ 2530.713] (EE) 3: /usr/bin/X (0x7f261ce05000+0x137c65) [0x7f261cf3cc65]
[ 2530.714] (EE) 4: /usr/bin/X (0x7f261ce05000+0x13a5d3) [0x7f261cf3f5d3]
[ 2530.714] (EE) 5: /usr/bin/X (0x7f261ce05000+0x15bc8f) [0x7f261cf60c8f]
[ 2530.714] (EE) 6: /usr/bin/X (mieqProcessDeviceEvent+0x1b5) [0x7f261cf8f385]
[ 2530.714] (EE) 7: /usr/bin/X (mieqProcessInputEvents+0xf8) [0x7f261cf8f4a8]
[ 2530.714] (EE) 8: /usr/bin/X (ProcessInputEvents+0x9) [0x7f261ce98789]
[ 2530.714] (EE) 9: /usr/bin/X (0x7f261ce05000+0x557d2) [0x7f261ce5a7d2]
[ 2530.715] (EE) 10: /usr/bin/X (0x7f261ce05000+0x4456a) [0x7f261ce4956a]
[ 2530.715] (EE) 11: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xed) [0x7f261adb076d]
[ 2530.715] (EE) 12: /usr/bin/X (0x7f261ce05000+0x448ad) [0x7f261ce498ad]
[ 2530.715] (EE)

10. Touch with one finger an other window. Focus is not transfered to the touched window. Expected behavior: The focus should be transfered to the touched window.
11. Touch with one finger inside nautilus window and do a drag gesture. Window is scrolling up and down and the above error lines are repeated at Xorg.0.log. Expected behavior: No error messages should be generated.
12. Remove the finger from the screen and move the mouse around the screen. Nautilus window is scrolling according to mouse movement. Expected behavior: no scrolling should happen at nautilus window.
13. Click with the mouse inside nautilus window. The rectangle selector appears and follows mouse movement. Expected behavior: A single click should not trigger the rectangle selector.

After these steps mouse and touch have a strange and unstable behavior: Mouse movement erroneously selects text without holding down the mouse button, clicks and touches at windows do not give focus, clicks and touches to buttons to not cause a press event, alt+tab is not functioning.

Touching with more fingers (3 or 4) produces a similar error with point number varying from 0 to 3 (e.g. (EE) [dix] N-trig DuoSense: unable to find touch point 2), and after that any gesture fails producing the above error messages.

I attach:
* geistest output
* lsb_release -a output
* lshw output
* lsinput output
* lspci -vnvn output
* mtdev-test output
* ntrig-calibrator output
* uname -a output
* cat /proc/version_signature output
* Xorg.0.log

I can perform any testing and reporting on Sony VAIO Duo 11 device. I have quite advanced technical skills on ubuntu, compiling, config files e.t.c. I would appreciate any guidance in order to investigate and finally solve this bug.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Konstantinos Togias (ktogias)]

Update:

bentiss from #xorg-devel IRC channel told that my device is not handled by hid-ntrig and is apparently handled by hid-multitouch in the kernel.

I followed the instructions at http://lii-enac.fr/en/architecture/linux-input/multitouch-howto.html#report and I attach the descriptors.txt file produced from report.py script.

[Konstantinos Togias (ktogias)]

Thanks to Benjamin Tissoir from lii-enac.fr, there is now an updated hid-multitouch kernel module for ubuntu 12.10. Thank you a lot Benjamin!

To install it follow the next steps from a terminal:
1. Follow the steps described in http://dinomite.net/blog/2007/setting-up-ubuntu-for-building-kernel-modules/ for setting up your machine for building kernel modules.
2. Install linux-headers package: sudo apt-get install linux-headers-`uname -r`
3. Move to your home: cd ~
4. git clone git://git.lii-enac.fr/linux-input/ubuntu-multitouch
5. cd ubuntu-multitouch
6. git checkout hid-multitouch-ubuntu-12.10
7. make
8. sudo make install
9. Reboot

I installed it and tested it on my Duo 11 with 12.10 Unity desktop:

* At kern.log I get the following mesages related to multitouch:
Feb 23 22:16:30 duo kernel: [ 4.903751] xhci_hcd 0000:00:14.0: WARN Event TRB for slot 2 ep 0 with no TDs queued?
Feb 23 22:16:37 duo kernel: [ 12.597436] hid-multitouch 0003:1B96:0F00.0004: usb_submit_urb(ctrl) failed: -1
Feb 23 22:16:37 duo kernel: [ 12.597457] hid-multitouch 0003:1B96:0F00.0004: timeout initializing reports
Feb 23 22:16:37 duo kernel: [ 12.597781] input: N-trig DuoSense as /devices/pci0000:00/0000:00:1a.0/usb1/1-1/1-1.1/1-1.1:1.0/input/input11
Feb 23 22:16:37 duo kernel: [ 12.598243] hid-multitouch

* One finger touches seem to work: Clicks on windows give focus, I can move windows with holding the title bar, I can put the cursor at a the touched point in a text area, I can scroll nautilus window with one finger.

* I have not managed to produce a double-touch for opening a folder in nautilus. After using touch with nautilus I can no longer double-click with the mouse, and right-click on a folder removes focus from it.

* When I put a second finger on the screen, at Xorg.0.log I no longer get any "(EE) [dix] N-trig DuoSense: unable to find touch point X" errors, but I get the same bug trace I provide at bug description "(EE) BUG: triggered 'if (!(event->device_event.flags & (1 << 5)))'".... , and no multi-finger guestures seem to be recognized. The behaviour is the sama as if I had only one finger on the screen (the first that touched the screen).

* Subsequent single finger touches continue to work and multifinger touches are recognized as single finger touches and continue to produce the "(EE) BUG: triggered 'if (!(event->device_event.flags & (1 << 5)))'" bug trace.

* The good thing is that with Benjamin's module there is no erroneous behavior of mouse, alt-tab etc after touching the screen with multiple fingers (except nautilus problems), so the user can use one finger touches with no problem.

I also tried with the X version from x.org-edgers ubuntu ppa. x.org-edgers ppa also installed the 3.7.0 Kernel, but I booted with 3.5.0-25 kernel in order to use Benjamin's module. The result was exactly the same as I described above.

I attach Xorg.0.log and kern.log with the new kernel module.

I would like to thank one more time Benjamin for his help on this.

[Konstantinos Togias (ktogias)]

UPDATE: The odd behavior in nautilus is described to those bugs:
https://bugs.launchpad.net/ubuntu-nexus7/+bug/1071922
https://bugs.launchpad.net/ubuntu/+source/xorg-server/+bug/1002788

So, it is not relevant with this bug.

[Konstantinos Togias (ktogias)]

I also attach the output of evtest command, run with Benjamin's kernel module. The output was produced by touching with one finger, removing, touching with 2, removing, 3, 4 and going on to 10 fingers.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael McCool (michael-d-mccool)]

I have followed these instructions, but have noted a few problems in addition the multitouch issue already mentioned:
1. After rebooting into Windows 8, touch NO LONGER WORKS. I tried shutting down and rebooting several times with the same result. It continues to work (for single touch) in Ubuntu.
2. The stylus does not work. It does not seem to be recognized as a wacom device. The stylus does work in Windows 8.
3. Double-tapping does not seem to result in a double-click.
4. The instructions for setting up kernel builds needs a few more uses of "sudo"

I have noticed flakey behaviour before in Windows related to touch being automatically disabled when the stylus is used then not getting reenabled afterwards. I don't know if that is relevant, since in that case it is fixed by rebooting Windows.

I'm wondering if the driver is leaving the hardware in an inconsistent state when shutting down and then Windows is unable to reinitialize it. Either way, rebooting does not seem to reset the issue.

[Michael McCool (michael-d-mccool)]

Shutdown, went to lunch for about 30mins, came back, rebooted into Windows. Now it works. Rebooted into Ubuntu, touch works. Rebooted into Windows, touch works. Shutdown, waited, rebooted into Ubuntu, touch works. Shutdown, waited, rebooted into Windows, touch works.

Huh.

Well, I wasn't hallicinating, but the problem seems to have gone away.

[Konstantinos Togias (ktogias)]

I confirm that the stylus is not recognized. We should open a new bug report for this. Also automatic screen rotation is not working. There should also be some event when the physical keyboard is revealed, that is not recognized (i remember that in win8 on screen keyboard was automatically disabled when the device was expanded to reveal keybord). Also the crucialtech trackpad gives random tapping events and i have not figured out a way to disable tapping... I have also some software bugs related to use with touch. Eg. firefox address and search bar lose focus after each key press on onscreen keyboard. All those should be reported in separate bugs. It's better to deal with one issue at a time.

[Josh Taylor (jtaylor231)]

Any updates for this? Would love to use the tablet with Ubuntu. Going to try Ubuntu 13.04 this weekend with it.

[Konstantinos Togias (ktogias)]

Hi Josh,

Benjamin has uploaded at the git repo metioned above versions of the module for kernels 3.6+ (just checkout branch hid-multitouch-3.6+). I have made some further tests with fedora 18 and ubuntu 12.10. There seems to be no bug with the kernel module, but I think Xorg is quite premature as regarding to multitouch resulting to various annoying bugs. In ubuntu unity I have managed to use single touch and 3+ fingers gestures (see https://wiki.ubuntu.com/Multitouch#Supported_Gestures), but two finger guestures confuse X (it's like X never catch the removal of the second finger and interpret all subsequent single finger touches as 2 finger ones).

I am going to also run some tests with 13.04, and post here a detailed report as soon as I get some time.

Please post here any observations, or comments when you test your duo with 13.04.

[Konstantinos Togias (ktogias)]

I have performed some tests with raring beta2 (linux 3.8.0-16-generic, X.Org X Server 1.13.3 - 2013-03-07 -, unity 6.12.0daily13.04.01-0ububtu1). The touch screen does not work out of the box. After the first touch it no longer responds to touches.

After installing Benjamin's kernel module (hid-multitouch-3.6+) the touchscreen works as expected most of the time. The bug "triggered 'if(!(event->device_event.flags & (1 << 5)))'" described at my very first post here, still apears from time to time while using the touch screen. I have not verified yet under what exact circumstances. I have also noticed that using the touch screen with some applications cause subsequent failure of getting a click event by tapping. E.g. In firefox tap on the search box. Then open onBoard keyboard. Try to type something in firefox search box with onBoard. After the first character firefox popups search suggestions that seem to take the focus from the text-box and no other characters are typed. After that tapping anywhere on the screen fails to produce a click. The problem may be solved by switching a couple of times between open windows/applications with alt+tab using the physical keyboard, or with logging out and in again. (Could this be related with the appearance of the bug error line in Xorg.0.log? ) The same problem exists with chromium web browser, when trying to tap on the options menu icon.

The good news is that nautilus is working good with the touch screen. Double taps are correctly caught and interpreted to double clicks.

After all, apart from the bug error in Xorg.0.log (can anyone tell what does it mean??) and the failure of click under the circumstances I described above, the touchscreen works fine with 13.04 at a basic level. The main problem seems to be the lack of proper multitouch suport of the main applications. Firefox does not scroll or zoom with guestures (drag n scroll pluggin can help for scrolling), the same with chromium. Evince seems to have some support for touch. I enjoyed reading a pdf with good accelerated scrolling, but no luck with zoom with pinch. Libreoffice writer is hard to use with multitouch (no scroll with drag).

I have read on the web about an ubuntu version oriented to tablets that is being prepared. From what I know it is tested against samsung galaxy tablets and has released some testing images for these devices (non x86 processors). Is there any luck that we will have some of the achievements of that project coming to the desktop x86 version of ubuntu?

[Konstantinos Togias (ktogias)]

I have been testing with Ubuntu 13.04 and 3.9 rc kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline for a couple of weeks. The kernel drivers seems to work good. No error messages or bug traces appear in logs. evtest properly reports BTN_TOUCH events all the time.

But I think I now have stuck on this unity(?) bug also reported on other devices: https://bugs.launchpad.net/ubuntu/+source/unity/+bug/1099289 : Randomly, after some time using the touchscreen unity stops registering click events on tapping. Click with tap continues to work on some application parts (e.g. gnome-terminal tabs, or unity shutdown/logout dialog), but launcher, window titlebar and most applications' windows cannot be clicked with tapping. I have also tested with gnome-shell and kde from 13.04 repositories and seem to have similar issues. After some time (especially when doing fast subsequent taps on the screen or trying to drag a scrollbar or type on the on-screen keyboard) the main components of the desktop environment (panels, dialogs etc) do not accept a click with tapping, while the cursor continues to move properly at the tapped point and evtest properly reports BTN_TOUCH events.

[Timur Lavrenti Kiyivinski (deathseclipse)]

Bug still present in Ubuntu 13.04 and has been present since 12.04 in my tests.

[Sergio López del Pozo (selopo)]

Installed 13.10 in my new Duo 11 and touch works just fine.

Now I'm trying to make the middle mouse button usable.