[enhancement] Autopilot testing needs to know screen coordinates of widgets on screen

[Gerry Boland]

On phablet devices, Autopilot needs to determine the absolute coordinates of any item on screen, in order to generate input events to interact with those items.

Most toolkits give AP the ability to calculate the coordinates of an item inside the surface, i.e. relative to the surface top-left corner, and then obtain the position of the window from the display server (X11 usually) to calculate the absolute position.

However Mir does not have an API to let its clients find the surface position relative to the screen. There is also a desire from the Mir team not to supply this information.

We need to discuss how to solve this problem in a way everyone is happy

[Thomi Richards (thomir-deactivatedaccount)]

A small addendum to the above:

autopilot-qt uses the Qt API to determine the coordinates of each widget in screen-space. Specifically, it uses QWidget::mapToGlobal to do the translation.

To my mind, since that API exists, and mir is supposed to support the Qt framework, the call should continue to work.

Gerry and I discussed a few other options, including exposing a dbus interface from unity8 that does the translation, but nothing I could come up with would be as straight forward and as clean as having the existing Qt API "just work".

It's also worth remembering that, while the current problem is with autopilot-qt, this is a public part of the Qt API, so it's possible that other applications in the future will be similarly broken by this change.

Cheers,

[Launchpad Janitor (janitor)]

This bug was fixed in the package qtubuntu - 0.60+14.10.20140808.1-0ubuntu1

---------------
qtubuntu (0.60+14.10.20140808.1-0ubuntu1) utopic; urgency=low

  [ Gerry Boland ]
  * Workaround for bug 1346633 broke QGuiApplication::topLevelWindows(),
    perform a different workaround. This workaround guesses the window
    geometry relative to the screen (mainly setting y=panelHeight),
    defining it using QPlatformWindow::setGeometry and then Qt uses it
    to calculate mapToGlobal internally (LP: #1351024, #1346633)
 -- Ubuntu daily release <email address hidden> Fri, 08 Aug 2014 14:10:40 +0000

[kevin gunn (kgunn72)]

assigning to chris as he's actually tackling

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:mir at revision None, scheduled for release in mir, milestone 0.9.0

[Launchpad Janitor (janitor)]

This bug was fixed in the package mir - 0.9.0+15.04.20141125-0ubuntu1

---------------
mir (0.9.0+15.04.20141125-0ubuntu1) vivid; urgency=medium

  [ Alberto Aguirre ]
  * New upstream release 0.9.0 (https://launchpad.net/mir/+milestone/0.9.0)
    - Enhancements:
      . New simpler API to configure and run a mir server.
      . The event loop is now based on GLib's main loop library instead of
        Boost.Asio.
      . For Android platforms, the server now sends buffer fence fds to its
        clients instead of potentially stalling the compositor thread waiting
        for them to be signalled.
      . New client debug interface to translate from surface to screen
        coordinates.
    - ABI summary: Servers need rebuilding, but clients do not;
      . Mirclient ABI unchanged at 8
      . Mircommon ABI bumped to 3
      . Mirplatform ABI bumped to 4
      . Mirserver ABI bumped to 27
    - Bug fixes:
      . Add a debug interface to translate from surface to screen coordinates
        (LP: #1346633)
      . Ensure a buffer requested by a surface is not delivered
        after the surface is deleted (LP: #1376324)
      . Overlays are not displayed onscreen in some positions (LP: #1378326)
      . Server aborts when an exception is thrown from the main thread
        (LP: #1378740)
      . Fix race causing lost alarm notifications (LP: #1381925)
      . Avoid lifecycle notifications racing with connection release
        (LP: #1386646)
      . Improve error checking and reporting for the client library
       (LP: #1390388)
      . Mir demo-shell now detects power button using proper Linux scan codes
       (LP: #1303817)
      . A prompt session with an invalid application pid should be an error
        (LP: #1377968)
      . When XDG_RUNTIME_DIR is defined but pointing to a non-existing
        directory use "/tmp" (LP: #1304873)
      . [regression] demo-shell bypass is not used on fullscreen surfaces if
        there are windowed surfaces behind (LP: #1378706)
      . Mir upgrade through dist-upgrade installs incorrect platform
        (LP: #1378995)
      . Fix Mir progressbar example using internal glibc defines(LP: #239272)
      . Stop the default_lifecycle_event_handler raising SIGHUP while
        disconnecting (LP: #1386185)
      . [regression] Mir fails to build with MIR_ENABLE_TESTS=OFF (LP: #1388539)
      . [regression] mir_demo_server_basic does not start (LP: #1391923)

  [ Ubuntu daily release ]
  * New rebuild forced
 -- Ubuntu daily release <email address hidden> Tue, 25 Nov 2014 17:49:24 +0000

[Olivier Tilloy (osomon)]

Note that this affects gallery-app autopilot tests. A recent change in the app makes it go fullscreen dynamically, thus invalidating the surface coordinates, and tests on krillin end up clicking off by 41px (the height of the top panel). See bug #1422418.

[Gerry Boland (gerboland)]

Note we've landed a hack to guess absolute positioning of surfaces for the phone. But a proper solution requires more thorough approach:
1. via the mir-client-debug API, a client can ask for the surface's absolute position
2. the server, if it is in debug mode, can reply with this information.
This requires patching qtubuntu, qtmir & unity8.

[Richard Huddie (rhuddie)]

A note to confirm this issue is affecting autopilot test automation on tablet devices, where the shell runs in landscape mode, but the display panel orientation is fixed portrait. When sending touch input co-ordinates relative to the shell orientation they must be transformed to the display orientation. But this requires knowledge of what the native display orientation is for the device being tested.

[Florian Boucault (fboucault)]

More workaround to include the low chrome mode at: https://code.launchpad.net/~fboucault/qtubuntu/fix_panel_hack_low_chrome