[enhancement] Autopilot testing needs to know screen coordinates of widgets on screen

Bug #1346633 reported by Gerry Boland
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Won't Fix
Fix Released
Chris Halse Rogers
mir (Ubuntu)
Fix Released
qtmir (Ubuntu)
qtubuntu (Ubuntu)
Fix Released

Bug Description

On phablet devices, Autopilot needs to determine the absolute coordinates of any item on screen, in order to generate input events to interact with those items.

Most toolkits give AP the ability to calculate the coordinates of an item inside the surface, i.e. relative to the surface top-left corner, and then obtain the position of the window from the display server (X11 usually) to calculate the absolute position.

However Mir does not have an API to let its clients find the surface position relative to the screen. There is also a desire from the Mir team not to supply this information.

We need to discuss how to solve this problem in a way everyone is happy

Tags: enhancement

Related branches

Revision history for this message
Thomi Richards (thomir-deactivatedaccount) wrote :

A small addendum to the above:

autopilot-qt uses the Qt API to determine the coordinates of each widget in screen-space. Specifically, it uses QWidget::mapToGlobal to do the translation.

To my mind, since that API exists, and mir is supposed to support the Qt framework, the call should continue to work.

Gerry and I discussed a few other options, including exposing a dbus interface from unity8 that does the translation, but nothing I could come up with would be as straight forward and as clean as having the existing Qt API "just work".

It's also worth remembering that, while the current problem is with autopilot-qt, this is a public part of the Qt API, so it's possible that other applications in the future will be similarly broken by this change.


tags: added: enhancement
summary: - AP needs absolute position of surface from Mir
+ [enhancement] Autopilot testing needs to know screen coordinates of
+ widgets on screen
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qtubuntu - 0.60+14.10.20140808.1-0ubuntu1

qtubuntu (0.60+14.10.20140808.1-0ubuntu1) utopic; urgency=low

  [ Gerry Boland ]
  * Workaround for bug 1346633 broke QGuiApplication::topLevelWindows(),
    perform a different workaround. This workaround guesses the window
    geometry relative to the screen (mainly setting y=panelHeight),
    defining it using QPlatformWindow::setGeometry and then Qt uses it
    to calculate mapToGlobal internally (LP: #1351024, #1346633)
 -- Ubuntu daily release <email address hidden> Fri, 08 Aug 2014 14:10:40 +0000

Changed in qtubuntu (Ubuntu):
status: New → Fix Released
Changed in autopilot:
status: New → Won't Fix
Revision history for this message
kevin gunn (kgunn72) wrote :

assigning to chris as he's actually tackling

Changed in mir:
importance: Undecided → High
assignee: nobody → Chris Halse Rogers (raof)
status: New → Triaged
Changed in mir:
milestone: none → 0.7.0
status: Triaged → In Progress
Changed in mir:
milestone: 0.7.0 → 0.8.0
Changed in mir:
milestone: 0.8.0 → 0.9.0
Changed in mir:
milestone: 0.9.0 → 0.8.0
Gerry Boland (gerboland)
Changed in qtmir:
status: New → Confirmed
importance: Undecided → High
Changed in mir:
milestone: 0.8.0 → 0.9.0
Revision history for this message
PS Jenkins bot (ps-jenkins) wrote :

Fix committed into lp:mir at revision None, scheduled for release in mir, milestone 0.9.0

Changed in mir:
status: In Progress → Fix Committed
Changed in mir (Ubuntu):
importance: Undecided → High
status: New → Triaged
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mir - 0.9.0+15.04.20141125-0ubuntu1

mir (0.9.0+15.04.20141125-0ubuntu1) vivid; urgency=medium

  [ Alberto Aguirre ]
  * New upstream release 0.9.0 (https://launchpad.net/mir/+milestone/0.9.0)
    - Enhancements:
      . New simpler API to configure and run a mir server.
      . The event loop is now based on GLib's main loop library instead of
      . For Android platforms, the server now sends buffer fence fds to its
        clients instead of potentially stalling the compositor thread waiting
        for them to be signalled.
      . New client debug interface to translate from surface to screen
    - ABI summary: Servers need rebuilding, but clients do not;
      . Mirclient ABI unchanged at 8
      . Mircommon ABI bumped to 3
      . Mirplatform ABI bumped to 4
      . Mirserver ABI bumped to 27
    - Bug fixes:
      . Add a debug interface to translate from surface to screen coordinates
        (LP: #1346633)
      . Ensure a buffer requested by a surface is not delivered
        after the surface is deleted (LP: #1376324)
      . Overlays are not displayed onscreen in some positions (LP: #1378326)
      . Server aborts when an exception is thrown from the main thread
        (LP: #1378740)
      . Fix race causing lost alarm notifications (LP: #1381925)
      . Avoid lifecycle notifications racing with connection release
        (LP: #1386646)
      . Improve error checking and reporting for the client library
       (LP: #1390388)
      . Mir demo-shell now detects power button using proper Linux scan codes
       (LP: #1303817)
      . A prompt session with an invalid application pid should be an error
        (LP: #1377968)
      . When XDG_RUNTIME_DIR is defined but pointing to a non-existing
        directory use "/tmp" (LP: #1304873)
      . [regression] demo-shell bypass is not used on fullscreen surfaces if
        there are windowed surfaces behind (LP: #1378706)
      . Mir upgrade through dist-upgrade installs incorrect platform
        (LP: #1378995)
      . Fix Mir progressbar example using internal glibc defines(LP: #239272)
      . Stop the default_lifecycle_event_handler raising SIGHUP while
        disconnecting (LP: #1386185)
      . [regression] Mir fails to build with MIR_ENABLE_TESTS=OFF (LP: #1388539)
      . [regression] mir_demo_server_basic does not start (LP: #1391923)

  [ Ubuntu daily release ]
  * New rebuild forced
 -- Ubuntu daily release <email address hidden> Tue, 25 Nov 2014 17:49:24 +0000

Changed in mir (Ubuntu):
status: Triaged → Fix Released
Changed in mir:
status: Fix Committed → Fix Released
Revision history for this message
Olivier Tilloy (osomon) wrote :

Note that this affects gallery-app autopilot tests. A recent change in the app makes it go fullscreen dynamically, thus invalidating the surface coordinates, and tests on krillin end up clicking off by 41px (the height of the top panel). See bug #1422418.

Revision history for this message
Gerry Boland (gerboland) wrote :

Note we've landed a hack to guess absolute positioning of surfaces for the phone. But a proper solution requires more thorough approach:
1. via the mir-client-debug API, a client can ask for the surface's absolute position
2. the server, if it is in debug mode, can reply with this information.
This requires patching qtubuntu, qtmir & unity8.

Revision history for this message
Richard Huddie (rhuddie) wrote :

A note to confirm this issue is affecting autopilot test automation on tablet devices, where the shell runs in landscape mode, but the display panel orientation is fixed portrait. When sending touch input co-ordinates relative to the shell orientation they must be transformed to the display orientation. But this requires knowledge of what the native display orientation is for the device being tested.

Revision history for this message
Florian Boucault (fboucault) wrote :

More workaround to include the low chrome mode at: https://code.launchpad.net/~fboucault/qtubuntu/fix_panel_hack_low_chrome

Michał Sawicz (saviq)
affects: qtmir → qtmir (Ubuntu)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers