nexus4 window mode mouse motion bounded incorrectly with external monitor

[kevin gunn]

when connecting a monitor to the Nexus4 during windowed mode, the device is put into landscape mode. the mouse motion appears to not have followed the orientation and is bounded by roughly portrait width instead of landscape width

The problem is that when unity8 configures cloning, it picks the intersection of all displays. While the input subsystem uses the bounding rectangle of all displays. Thus it allows the cursor to move around in (1200x1920) while the only surface is 1080x1920. So there is a dead zone between the display extent and the surface extent. When the cursor hits that area the SurfaceInputDispatcher has no surface underneath, so it decides to not send any further mouse input to Unity8.

We can fix that by extending the behaviour of display_input_region.cpp to use the intersection of displays when necessary.

[Daniel van Vugt (vanvugt)]

Might be bug 1313610, or closely related.

[Daniel van Vugt (vanvugt)]

Yeah pretty sure it's the same as bug 1313610. We've never updated our cursor bounds correctly on rotation...

[kevin gunn (kgunn72)]

unduping for the moment, after some discussion with anpok, while this is similar we're not totally convinced it's the same

@duflu also, your are welcome please try this - specifically notice that if you move the mouse completely horizontal, the mouse will stick in a position that is inconsistent (e.g. sometimes sticks at a position further left or shorter left from the right edge than the previous time)
also, when the mouse "sticks" to a position while moving completely horizontal to the right, keep moving horizontal to the right, then move completely horizontal back to the left....the distance you travel back to the left is ~ the same before the mouse is "picked up" and "unstuck" giving it a feeling the position is changing for the mouse, however not rendering correctly.

note this is silo 0 where unity8 is doing the render - using the mir relative mouse events
to install
flash nexus4 with --channel=ubuntu-touch/rc-proposed/ubuntu
then install silo 0, unfortunately the silo is dirty holding back unity8 (atm i think) - so even if you install using ctrain tools you'll need to dist-upgrade
so....citrain device-upgrade 0 <device pin/password> , then apt-get update, apt-get dist-upgrade

[kevin gunn (kgunn72)]

more information i should add - this is not seen on nexus7, which in windowed mode is always in landscape and doesnt rotate
on nexus4, if a second monitor is not attached, this is not seen, even in windowed mode (which rotates to landscape, e.g. just connect a bt mouse will put into windowed mode)
but once a monitor is connected to the nexus 4, this can be seen

[Daniel van Vugt (vanvugt)]

Yeah there may be some common things to fix with bug 1313610 but possibly some things different here.

[Daniel van Vugt (vanvugt)]

OK, I can see one problem already. While Unity8 does its own screen rotation without notifying Mir, Mir cannot do anything to help it update the cursor bounds. Because Unity8 is not telling Mir that the display has rotated, Mir will keep the old cursor bounds...

Portrait:

$ mirout /var/run/mir_socket
Connected to server: /var/run/mir_socket
Card 0: Max 2 simultaneous outputs
Output 0: Card 0, LVDS, connected, 768x1280+0+0, used, on, 61mm x 102mm (4.7"), normal
     768x1280 60.00*+
Output 1: Card 0, DisplayPort, disconnected

Landscape (incorrectly shows orientation still "normal"):

$ mirout /var/run/mir_socket
Connected to server: /var/run/mir_socket
Card 0: Max 2 simultaneous outputs
Output 0: Card 0, LVDS, connected, 768x1280+0+0, used, on, 61mm x 102mm (4.7"), normal
     768x1280 60.00*+
Output 1: Card 0, DisplayPort, disconnected

So if Unity8 is going to keep doing rotation independently of Mir, it either needs to find a way to update the Mir display config without making Mir actually apply that display config, or Unity8 do mouse movement entirely within itself using relative motion.

[Daniel van Vugt (vanvugt)]

Oh... is there another Mir bug too? Is Mir failing to provide continuous relative motion data without hitting invisible bounds?

[Daniel van Vugt (vanvugt)]

Aha! Confirmed on arale with a USB mouse and mir-demos.

Relative motion events usually work continuously, and come in even after the cursor hits the screen boundary. However Mir stops sending relative motion events if you've rotated the screen (touch+volume key in mir_proving_server) and the cursor happens to be outside of the original non-rotated dimensions. That's a Mir bug.

[Daniel van Vugt (vanvugt)]

My mistake. Not a Mir bug. I was looking at MIR_CLIENT_INPUT_RECEIVER_REPORT=log and expecting the client to receive motion events after the cursor had left the window, which was wrong.

Using a more correct test where the gesture starts in the app window (click but don't release), then you can see the client continues to receive relative motion data even after hitting the edge of the screen:

[1440559624.200713] <DEBUG> input-receiver: Received event:pointer_event(when=1440559624199932000 (0.685269 ms ago), from=6, motion, button_state=1, x=1919, y=1151, dx=1, dy=1, vscroll=0, hscroll=0, modifiers=1)

[1440559624.208718] <DEBUG> input-receiver: Received event:pointer_event(when=1440559624207936000 (0.687115 ms ago), from=6, motion, button_state=1, x=1919, y=1151, dx=0, dy=1, vscroll=0, hscroll=0, modifiers=1)

[1440559624.377396] <DEBUG> input-receiver: Received event:pointer_event(when=1440559624376062000 (1.160346 ms ago), from=6, motion, button_state=1, x=1919, y=1151, dx=0, dy=1, vscroll=0, hscroll=0, modifiers=1)

So Mir seems to be working properly. It even keeps sending relative motion events after the cursor has hit the edge.

[Daniel van Vugt (vanvugt)]

Please try this:

   restart unity8 MIR_CLIENT_INPUT_RECEIVER_REPORT=log
   tail -f ~/.cache/upstart/unity8.log

and see if Unity8 actually is still receiving input events with relative data (dx, dy) when the problem occurs.

[Daniel van Vugt (vanvugt)]

Please try comment #10 as that should help us to identify if the problem is on the Mir side or Unity8.

[Daniel d'Andrada (dandrader)]

I will investigate the claim that qtmir is still receiving relative mouse movement even when you see the cursor hitting an invisible boundary

[Daniel d'Andrada (dandrader)]

Mir keeps sending pointer events after it hits a boundary only if you're using the primary display. If you connect and external monitor, it doesn't work like that anymore.

Steps to reproduce:
- Flash an N7 and install silo0 in it.
- Add "initctl set-env --global QT_LOGGING_RULES=qtmir.mir.input.debug=true" to /usr/share/upstart/sessions/unity8.conf
- sudo restart lightdm
- in a phablet terminal: tail -f ~/.cache/upstart/unity8.log
- connect a bluetooth mouse and check the output, you should see a lot of entries like "qtmir.mir.input: Received MirPointerEvent(x=971,y=573,relative_x=-1,relative_y=0)"
- see that the mouse goes through the entire screen and you always get mouse events even when the pointer reaches a boundary
- now conenct an external monitor and see what happens

expected outcome:
mouse events keep coming like before

actual outcome:
you have invisible boundaries since mouse events stop coming when the pointer reaches some width and height boundary.

Note that the cursor you see on the screen is drawn by unity8 and it's driven solely by the relative movement in MirPointerEvents.

[Daniel d'Andrada (dandrader)]

To get the debug output you've to make sure your qtmir is at least lp:~unity-team/qtmir/silo0 revision 369 ("QtEventFeeder: log the pointer events it gets from Mir")

[Daniel van Vugt (vanvugt)]

Can you double check using the report in comment #10? It's still theoretically a small possibility that the bug lies in unity8 till that's confirmed.

[Daniel van Vugt (vanvugt)]

Dropped back to High (https://wiki.ubuntu.com/Bugs/Importance)

[Daniel d'Andrada (dandrader)]

@duflu

There you go:
http://paste.ubuntu.com/12213056/

When you hit the invisible boundary on the external monitor nothing comes up in the log. Neither from mir nor qtmir. Happy now? :)

[Daniel d'Andrada (dandrader)]

Now we know what the problem is, and it will be gone once qtmir+unity8 start using one surface per display instead of cloning displays as it's currently happening.

[kevin gunn (kgunn72)]

yep, like daniel said we'll lower priority on this since qtmir multimonitor branch is getting cleaned up and landed.

[Daniel van Vugt (vanvugt)]

Comment #18 kind of makes sense.

In trying to replicate the bug using just Mir demos I found one way to do it: Have a client (which in our case is Unity8 itself) that fails to resize to cover all displays. As it fails to resize to cover all displays, the client (Unity8 shell) will fail to get motion events that are outside of its window.

So this problem can be solved in Unity8 by resizing the Unity8 Mir surface to cover all displays, or it can be solved more reliably with a Mir enhancement (which we plan on already) --> implement pointer grabs so that the cursor location does not ever stop the owning surface from receiving pointer events.

Either way, seems like Mir is working correctly. In the absence of pointer grab support, clients (including Unity8) stop getting pointer events when the cursor leaves their window.

[Daniel van Vugt (vanvugt)]

Actually there might still be a related bug in Mir. Seems unlikely, but not really sure yet.

[Daniel van Vugt (vanvugt)]

I'm fairly sure this bug is invalid for Mir per comment #20. Although pointer grabs in Mir would help to solve the problem, that's really an enhancement for later. In comment #21 I was just being cautious in case anyone can correct me.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Daniel van Vugt (vanvugt)]

I've found the same kind of bug, but no external monitor required --> bug 1502805.

[kevin gunn (kgunn72)]

this is fixed in silo 22 which should land soon

[Daniel d'Andrada (dandrader)]

Actually input still gets in a pretty sorry state when an external monitor is connected.
But waiting for silo 22 to land before raising the issue to mir guys. Will make for easier debugging and communication.

[kevin gunn (kgunn72)]

seems corrected in latest image
channel: ubuntu-touch/rc-proposed/ubuntu-pd
version version: 54
version ubuntu: 20151026
version device: 20150911
version custom: 20150929-2-vivid