[regression] mouse pointer support on emulator is broken

Bug #1517597 reported by Dave Morley on 2015-11-18
44
This bug affects 8 people
Affects Status Importance Assigned to Milestone
Canonical System Image
Critical
kevin gunn
Client Developer Experience
Undecided
Unassigned
Mir
Fix Released
Critical
Kevin DuBois
0.18
Won't Fix
Critical
Unassigned
0.19
Won't Fix
Critical
Unassigned
0.20
Fix Released
Critical
Kevin DuBois
android (Ubuntu)
Critical
Andreas Pokorny
mir (Ubuntu)
Critical
Unassigned

Bug Description

STEPS:
1. Flash with sudo ubuntu-emulator create --channel ubuntu-touch/rc/ubuntu rc-test
2. Wait for it to build
3. Run with ubuntu-emulator run rc-test
4. Starts up but the mouse never moves meaning you can't leave the first page of the welcome wizard

EXPECTED:
I expect the emulator cursor to follow the mouse cursor and be able to click on things

ACTUAL:
Emulator cursors stay in the top left preventing movement.

As noted below F6 toggles the input mode.
The pointer movement is quite slow and jerky.
You also can no longer initiate an edge swipe even in trackball mode.

This is a major regression from how the emulator used to perform.

Related branches

Dave Morley (davmor2) on 2015-11-18
tags: added: qa-regression-testing
Changed in canonical-devices-system-image:
assignee: nobody → kevin gunn (kgunn72)
importance: Undecided → High
milestone: none → ww02-2016
tags: added: regression-proposed
tags: added: lt-blocker lt-category-visible
kevin gunn (kgunn72) on 2015-11-19
Changed in mir (Ubuntu):
assignee: nobody → Andreas Pokorny (andreas-pokorny)
importance: Undecided → High

Here are actually three problem at work:
 * The input device exposed inside the emulator /dev/input/event0 only emit absolute coordinate events. We recently experienced something similar with other incarnation of qemu in kvm: https://bugs.launchpad.net/mir/+bug/1489522 ( in that bug two mouse devices are available, the one that claims to expose relative events never emits them) The actual cause might be in the emulator integration itself.
 * The input device only emits those event on mouse button down. WIthout pressing a mouse button the evdev device remains silent. Another emulator problem?
 * The android input stack used in mir does not care about absolute mouse movement events. So even on mouse down it does not handle those events. The upcomig mir will, since it will default to mir-platform-input-evdev. So this will disappear with 0.18

At the moment you can see the mouse moves when you change to the new stack:
apt-get update && apt-get install mir-platform-input-evdev3
then edit /usr/share/ubuntu-touch-session/usc-wrapper
and add
 --platform-input-lib /usr/lib/i386-linux-gnu/mir/server-platform/input-evdev.so.3
to the call to unity-system-compositor

Or is the intention of the emulator to simulate a touch screen - hence the input events only appear on press?

After some digging inside the code of the emulator.. The behaviour described above seems to be intended - at least to some degree.

To move the mouse the user has to press F6 to toggle into trackball mode and use the mouse cursor. And it should work as expected.

Pat McGowan (pat-mcgowan) wrote :

is this not a valid bug then? or does the default need to change?

Changed in canonical-devices-system-image:
status: New → Incomplete

After trying around a bit more... If we intend to use the emulator in the future we might want to tweak which input devices are exposed the guest system. RIght now there is a single odd abs-relative-touchscreen-mouse hybrid. For odd reasons it was treated as touch screen in the past (we didnt really differ with the android stack: everything not a key event is a motion event). With libinput it will be seen as a mouse that either emits absolute or relative coodinates. So we will lack the capability to simulate touch input.

Additionally the relative mouse mode works very badly. It seems to skip a lot of mouse movement or they are scaled down?

For either of the problems we might have to change the emulator window code. Which I believe we take from here:

https://android.googlesource.com/platform/external/qemu/+/master/android/skin/

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in mir (Ubuntu):
status: New → Confirmed
Changed in phablet-tools (Ubuntu):
status: New → Confirmed
Łukasz Zemczak (sil2100) wrote :

It is a valid point, I suppose we should expose a setting that would allow the user to switch between a mouse input and touch input simulation. We might need a separate bug for that though.

That being said, is the emulator working now as before? Would like QA to validate if it's at least better now and allows to continue testing the few emulator bits we do.

Dave Morley (davmor2) wrote :

Emulator comes up, cursor moves however it doesn't match the position of the mouse cursor so clicks are failing and I can't get off the first page of the welcome wizard.

Pat McGowan (pat-mcgowan) wrote :

@Dave did you toggle the mode with F6?

summary: - Ubuntu-emulator is broken on rc image 3
+ mouse pointer support on emulator is broken
description: updated
tags: added: input regression
Changed in mir:
milestone: none → 0.19.0
importance: Undecided → High
summary: - mouse pointer support on emulator is broken
+ [regression] mouse pointer support on emulator is broken
tags: added: emulator
Changed in mir:
status: New → Confirmed
Changed in mir (Ubuntu RTM):
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Andreas Pokorny (andreas-pokorny)
Changed in mir:
assignee: nobody → Andreas Pokorny (andreas-pokorny)
Michał Sawicz (saviq) wrote :

FWIW do we have an idea how this all *should* work?

Having a mouse device in a windowed emulator feels wrong unless we have the cursor contained/grabbed, otherwise you won't be able to push against the edges for launcher/spread etc.

IMO by default the device should mimic touch events, with a switch to mouse events that would contain the cursor in the emulator. This way you can easily switch between windowed and touch scenarios, too.

kevin gunn (kgunn72) wrote :

Right - so leaving this comment here to prevent me from asking anpok again ;)

The solution believed to be correct is to modify the Android emulator code to properly emulate a touch device (which is actually a mouse). Not much we can do on our side.

Changed in android (Ubuntu):
assignee: nobody → Andreas Pokorny (andreas-pokorny)
Changed in mir:
status: Confirmed → Opinion
no longer affects: mir/0.18
Changed in android (Ubuntu):
importance: Undecided → High
no longer affects: mir (Ubuntu RTM)
Changed in mir (Ubuntu):
status: Confirmed → Opinion
Changed in mir:
assignee: Andreas Pokorny (andreas-pokorny) → nobody
Changed in mir (Ubuntu):
assignee: Andreas Pokorny (andreas-pokorny) → nobody
Daniel van Vugt (vanvugt) wrote :

I think the emulator is correct, albeit weird. Qemu provides a tablet virtual device (so you don't get two mouse pointers and can enter/leave the VM window freely), and it works well with Ubuntu and Windows. And it sounds like it worked well with Mir till recently.

The solution is not to modify the emulator, because the emulator is correctly designed to emulate a tablet. Much like that of a stylus-based tablet where the mouse cursor does follow your hovers if you're close enough, or you can jump to a new location. This is a mature established feature of Qemu that's been there for many years. So let's just fix Mir/libinput.

Daniel van Vugt (vanvugt) wrote :

P.S. I have a real-life tablet device (X220t) of this sort and have tested it with Mir over the years. You can see the bugs here:
https://bugs.launchpad.net/mir/+bugs?field.tag=wacom

Although I suspect it may not work with Mir as well as it used to when I logged those bugs.

Changed in mir:
status: Opinion → Confirmed
Changed in mir (Ubuntu):
status: Opinion → Confirmed
Daniel van Vugt (vanvugt) wrote :

To elaborate... what you see on a "traditional" stylus-based Wacom tablet is probably what Qemu is implementing. That means:

* Stylus is near (within a few centimetres) but not touching the screen: Mouse cursor visible and follows the pen (but ideally shows something different to an arrow, like a pencil cursor).

* Stylus touches the screen: That's a touch or primary button-down event, and button up event for release.

* Stylus moves away from the screen: No cursor is visible.

I think that's what Qemu is trying to emulate and it's probably a good model for our controlling a virtual touch screen with a mouse too. So we should fix libinput and/or Mir to work in this environment. Obviously it did until recently.

Changed in canonical-devices-system-image:
status: Incomplete → Confirmed
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in android (Ubuntu):
status: New → Confirmed
Changed in canonical-devices-system-image:
importance: High → Critical
kevin gunn (kgunn72) on 2016-01-14
Changed in mir:
importance: High → Critical
Changed in android (Ubuntu):
importance: High → Critical
Changed in mir:
status: Confirmed → Opinion
no longer affects: phablet-tools (Ubuntu)
Changed in mir (Ubuntu):
status: Confirmed → Opinion
importance: High → Critical
tags: added: lt-important
removed: lt-blocker
Changed in mir:
milestone: 0.19.0 → none
status: Opinion → Invalid
Changed in mir (Ubuntu):
status: Opinion → Invalid

This is not a stylus emulation.. Touch screen and Mouse devices overlap in the axes they use. SInce mice with absolute coordinates are allowed too (i.e VMWare Mouse Emulation). A device that tries to mimic both will run into problems, since there are differences in the press/release behavior. If you then go on and add multitouch axes to the mix it will get even worse, because suddenly the difference between a touchscreen a touchpad and a mouse is lost. A device that changes its behavior without notifying user space is not really helping here either. The android-qemu fork could solved that in various ways. Especially not cramping everything into a single device would have helped a lot. But I will focus on getting the touch screen emulation working again. It does now locally after removing the relative mouse events, but still the edge swipes do not work. Need to look into that further.

Changed in canonical-devices-system-image:
milestone: ww02-2016 → ww08-2016
Jamie Strandboge (jdstrand) wrote :

I'm told that this bug is a promotion blocker for stable channel emulator images. With vivid going EOL next week, the security team will need to have working emulator images for at least stable and rc-proposed to properly support touch devices since everyone doesn't have devices for testing. Can we get this fixed soon?

kevin gunn (kgunn72) wrote :

It's not off our radar i promise

kevin gunn (kgunn72) on 2016-02-11
no longer affects: android (Ubuntu RTM)
Daniel van Vugt (vanvugt) wrote :

This one is more opinion than invalid for the Mir project. Indeed the bug started happening after a change in the Mir project.

Changed in mir:
status: Invalid → Opinion

I wanted to share some status update, the emulated touchscreen troubles have been resolved. Now ui elements fail to detect any gestures, the display claims its phsyical width, height and dpi to be 0.

Mir report 0x0 is the actually something to be fixed in mir to get that bug resolved.

Changed in mir:
status: Opinion → Confirmed

Mir problem is solved, currently not yet integrated into upcoming 0.20.
quemu fix is on the way:
https://code-review.phablet.ubuntu.com/#/c/431/

Changed in mir:
assignee: nobody → Kevin DuBois (kdub)
PS Jenkins bot (ps-jenkins) wrote :

Fix committed into lp:mir at revision 3314, scheduled for release in mir, milestone 0.21.0

Changed in mir:
status: Confirmed → Fix Committed
Changed in mir:
milestone: none → 0.21.0
Daniel van Vugt (vanvugt) wrote :

We should probably consider backporting this fix to 0.20.0

Changed in mir (Ubuntu):
status: Invalid → Triaged
kevin gunn (kgunn72) wrote :

> We should probably consider backporting this fix to 0.20.0
totally agree

Daniel van Vugt (vanvugt) wrote :

Fix committed (backported) to lp:mir/0.20 at revision 3319. Scheduled for release in Mir 0.20.0

Launchpad Janitor (janitor) wrote :
Download full text (3.3 KiB)

This bug was fixed in the package mir - 0.20.0+16.04.20160219-0ubuntu1

---------------
mir (0.20.0+16.04.20160219-0ubuntu1) xenial; urgency=medium

  [ Alan Griffiths ]
  * New upstream release 0.20.0 (https://launchpad.net/mir/+milestone/0.20.0)
    - ABI summary: Only servers need rebuilding;
      . mirclient ABI unchanged at 9
      . mirserver ABI bumped to 38
      . mircommon ABI unchanged at 5
      . mirplatform ABI unchanged at 11
      . mirprotobuf ABI unchanged at 3
      . mirplatformgraphics ABI bumped to 8
      . mirclientplatform ABI unchanged at 4
      . mirinputplatform ABI unchanged at 5
    - Enhancements:
      . Allow screencasting to create a virtual output (for Miracast)
      . Separate the protocol version number from the client API version macros.
        They're not meant to be related concepts.
      . Add UBSanitizer to the list of build types.
      . logging: Human readable timestamps in DumbConsoleLogger.
      . examples: AdorningDisplayBufferCompositor::composite() no long ignores
        output boundaries and occlusions.
      . examples: Add -a <app name> option to eglapps.
      . common, client: a more flexible way to probe modules: once we've found
        a good current platform we don't even try to load an older one.
      . Fix build and test run with CMAKE_BUILD_TYPE=ThreadSanitizer (missing
        locks).
      . Add MIR_USE_LD_GOLD build option.
    - Bug fixes:
      . unity-system-compositor crashed with std::runtime_error in
        mir::compositor::CompositingFunctor::wait_until_started() from
        usc::MirScreen::set_screen_power_mode (mir_power_mode_on)
        (LP: #1528384)
      . Phone not usable while a call comes in - followed by "restart"
        (LP: #1532607)
      . ui freezes when simultaneously moving mouse & plug/unplug hdmi
        (LP: #1538632)
      . Mir fails to build on xenial today: android_graphic_buffer_allocator.h
        fatal error: hardware/hardware.h: No such file or directory
        (LP: #1539338)
      . [mali] egl_demo_client_flicker has graphics corruption on android
        (LP: #1517205)
      . [testsfail] Intermittent failure in
        TestClientCursorAPI.cursor_passed_through_nested_server (LP: #1525003)
      . [android] External monitor slows rendering (LP: #1532202)
      . Display::create_gl_context may create context with incorrect attributes
        (LP: #1539268)
      . unity-system-compositor locked up in __libc_do_syscall() (LP: #1543594)
      . NestedServer.client_sees_set_scaling_factor intermittent failure
        (LP: #1537798)
      . [android] External monitor slows rendering - part 2 (LP: #1535894)
      . scene: make sure not to set the swapinterval to 0 when an independent
        stream is created. The default should be 1 (like the stream created as
        part of surface creation).
      . Track the displays plugged state to avoid reporting configurations in
        case they are unplugged (LP #1531503). [Cherrypicked from 0.21]
      . mouse pointer support on emulator is broken (LP: #1517597).
        [Cherrypicked from 0.21]
      . move an android-only test that ended up in tests/unit-tests/graphics.
        (LP: ##154...

Read more...

Changed in mir (Ubuntu):
status: Triaged → Fix Released
Changed in mir:
status: Fix Committed → Fix Released
Changed in canonical-devices-system-image:
status: Confirmed → Fix Committed
Changed in mir:
status: Fix Released → Fix Committed
Daniel van Vugt (vanvugt) wrote :

This bug got separate (diverged) fixes in Mir 0.21 and 0.20. So that's why it has two separate tasks.

Simon Fels (morphis) on 2016-03-07
Changed in android (Ubuntu):
status: Confirmed → In Progress
Simon Fels (morphis) wrote :

New release for the android package is in progress now at https://requests.ci-train.ubuntu.com/#/ticket/1081

kevin gunn (kgunn72) wrote :

this is now showing landed

Changed in android (Ubuntu):
status: In Progress → Fix Released
Changed in mir:
status: Fix Committed → Fix Released
status: Fix Released → Fix Committed
Changed in mir:
status: Fix Committed → Fix Released
Changed in canonical-devices-system-image:
status: Fix Committed → Fix Released
Changed in canonical-developer-experience:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers