[regression] Window resizing is very choppy and stutters (border does not smoothly follow the mouse)

[Daniel van Vugt]

Just tried Unity8 desktop for the first time since October and found some things don't work as well as before. One of them is window resizing, which is now every choppy and stuttering.

Please also see bug 1540702 so as to not confuse the two.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: unity8 8.11+16.04.20160122-0ubuntu1
ProcVersionSignature: Ubuntu 4.3.0-7.18-generic 4.3.3
Uname: Linux 4.3.0-7-generic x86_64
ApportVersion: 2.19.4-0ubuntu1
Architecture: amd64
Date: Thu Jan 28 18:39:38 2016
InstallationDate: Installed on 2015-12-03 (56 days ago)
InstallationMedia: Ubuntu 16.04 LTS "Xenial Xerus" - Alpha amd64 (20151202)
SourcePackage: unity8
UpgradeStatus: No upgrade log present (probably fresh install)
upstart.unity8-dash.log: Attempted to deliver an event to a non-existent window, ignoring.

[Albert Astals Cid (aacid)]

resizing of any window or the dash?

[Daniel van Vugt (vanvugt)]

It's the dash. I don't appear to have any other apps with which to open other windows yet.

That said, the dash occasionally freezes and during that time resizing its window becomes smooth :)

[Albert Astals Cid (aacid)]

Then it's https://code.launchpad.net/~aacid/unity8/dash_backgroud_source_size_rework/+merge/276157 probably

[Daniel van Vugt (vanvugt)]

I can confirm bug 1532974 is an issue but it's a separate bug. This bug is about the window border failing to follow the mouse during resizing. It fails to move, but occasionally jumps.

It feels like we've broken the basic rule of display servers in that the server must never wait for a client.

[Gerry Boland (gerboland)]

Correct window resizing has not been implemented yet. What's currently in place is indeed suboptimal.

[Launchpad Janitor (janitor)]

This bug was fixed in the package unity8 - 8.11+16.04.20160129-0ubuntu1

---------------
unity8 (8.11+16.04.20160129-0ubuntu1) xenial; urgency=medium

  [ Albert Astals Cid ]
  * Audio Preview Button: Don't gain focus on press (LP: #1517830)
  * Better attempt at sanitization
  * Better looking application scopes for wide screens
  * Call printSettings correctly
  * Card test: Limit the precision we ask for doubles to be equal
  * Convert while into tryCompareFunction
  * Do not use the same filepath parameter for all the card creator
    createQmlObject calls (LP: #1536379)
  * Fix some test shells being unstable in the armhf builders
  * Make revealByEdgePush more stable
  * More reliable compares
  * Move the mouse over the bfb before waiting for rendering
  * Remove the DDAs constrains for every test
  * Set the background source size to the max we'll need + new
    background (LP: #1539011)

  [ Andrea Cimitan ]
  * Dynamically change click scope card size according to size (LP:
    #1536745)
  * Should fix lazyimage test failures on fast machines

  [ CI Train Bot ]
  * Fix card tests on the phone
  * Update translation template

  [ Daniel d'Andrada ]
  * Desktop stage rotates only to follow screen orientation (LP:
    #1515977)

  [ Lukáš Tinkl ]
  * Hide/reveal the mouse pointer on touch/mouse events renamed:
    plugins/Utils/HomeKeyWatcher.cpp =>
    plugins/Utils/WindowInputMonitor.cpp plugins/Utils/HomeKeyWatcher.h
    => plugins/Utils/WindowInputMonitor.h
    tests/mocks/Utils/HomeKeyWatcher.qml =>
    tests/mocks/Utils/WindowInputMonitor.qml
    tests/plugins/Utils/HomeKeyWatcherTest.cpp =>
    tests/plugins/Utils/WindowInputMonitorTest.cpp

  [ Michael Terry ]
  * Synchronize the mouse-cursor-speed and touchpad-cursor-speed
    properties with USC. (LP: #1536383)

  [ Michael Zanetti ]
  * Add InputMethod to secondary screen
  * Add a virtual Touchpad while external screen is connected
  * Make big tablets switch to windowed mode by plugging a kbd only (LP:
    #1535399)
  * Prevent alt key presses from being dispatched to the app on alt+tab
    (LP: #1532862)

  [ Michał Sawicz ]
  * Make qmluitests.sh more robust and allow parallel test execution
  * Skip SessionBackendTest::testLogin1Capabilities until we talk to a
    mock instead (LP: #1532112, #1532113)

 -- Michał Sawicz <email address hidden> Fri, 29 Jan 2016 11:53:27 +0000

[Daniel d'Andrada (dandrader)]

That was a design decision (made by developers, didn't actually go through the design team).

First we were ignoring the mir surface size when resizing, so if it didn't match or lagged behind the QML window size, the surface would get stretched or squeezed to fit inside it.

Now the QML window size *always* match the Mir surface size. So the resize "animation" will be as smoothly as the client application is able to redraw its mir surface.

I think we need design input on how the window resize UX should look like. Ie, how to deal with applications that are slow to relayout/redraw and still have a smoothly "resize animation". Should we display some sort of size hint such as that translucent orange rectangle in Unity 7?

[Daniel van Vugt (vanvugt)]

"Now the QML window size *always* match the Mir surface size. So the resize "animation" will be as smoothly as the client application is able to redraw its mir surface."

See comment #5 :) The basic rule of display servers is that you should never wait for clients. Never.

The solution to this problem is what we use in the Mir demo servers, which is the same as what you will find in Unity7, Microsoft Windows, and ChromeOS at least. That is:
  (1) The server never waits for the client to do anything. Just acts on the latest information it has.
  (2) The server chooses texture coordinates such that the client's pixels always appear square.
  (3) Per number 2, if the client is lagging a bit, that's fine. Just fill in the blank space or truncate.

This is what you will find Mir demo servers do, as well as Unity7, Microsoft Windows (including Windows 10) and ChromeOS. I've verified these all recently.

Unfortunately I did propose a fix to this problem in August 2015. And although it got rejected, I thought we had landed something equivalent, but apparently not. (https://code.launchpad.net/~vanvugt/qtmir/unstretch/+merge/268724)

[Daniel van Vugt (vanvugt)]

Also note the current window resizing implementation in QtMir is creating additional lag that shouldn't be there. Waiting for the client to update simply takes too long. As a result, the window edges lag behind the mouse cursor. However since Unity8 renders its own mouse cursor there should be _zero_ visible lag between the cursor and the screen, as demonstrated with:

mir_demo_client_target -m /var/run/user/1000/mir_socket -- --desktop_file_hint=unity8

(after you resize it... seems there's a new QtMir bug there)

[Daniel van Vugt (vanvugt)]

Or you can just try dragging the window by its title bar. That shows the zero lag* that we should be seeing during resizing too.

* "zero lag" visually between the mouse cursor and the thing being dragged. Not zero lag between the physical mouse and the screen... we're still working on that.

[Daniel d'Andrada (dandrader)]

On 03/02/2016 00:07, Daniel van Vugt wrote:
> [...] if the client is lagging a bit, that's fine. Just fill in the blank space or truncate.

That's a UI design decision. I personally advocated for that approach,
but got outvoted in my team, which preferred the implementation we have
now (which I explained in my previous comment).

[Daniel d'Andrada (dandrader)]

On 03/02/2016 03:27, Daniel van Vugt wrote:
> Also note the current window resizing implementation in QtMir is
> creating additional lag that shouldn't be there. Waiting for the client
> to update simply takes too long. As a result, the window edges lag
> behind the mouse cursor. However since Unity8 renders its own mouse
> cursor there should be _zero_ visible lag between the cursor and the
> screen, as demonstrated with:
>

The "laggy and choppy" resize for apps that are slow to redraw/relayout
is all Unity 8, not QtMir. I can implement it the way you want (making
window border follow mouse and filling the empty space between it and
the surface with white for instance) without changing a single line in
QtMir.

[Daniel van Vugt (vanvugt)]

Filling the gap with white would be an improvement, but not ideal. Ideal I think is to 'clamp' the client texture to the edge, so you get whatever colour the edge pixels of the client's surface is, without stretching the rest of the surface. A good example of this is in the Mir demo servers' resizing implementation (GL renderer), or:
https://code.launchpad.net/~vanvugt/qtmir/unstretch/+merge/268724

[Daniel d'Andrada (dandrader)]

On 05/02/2016 08:26, Daniel van Vugt wrote:
> Filling the gap with white would be an improvement, but not ideal. Ideal I think is to 'clamp' the client texture to the edge, so you get whatever colour the edge pixels of the client's surface is, without stretching the rest of the surface. [...]
>

Or that (falls into the same category). Also offered that
option/variation back then, still outvoted by the current implementation.

[Daniel van Vugt (vanvugt)]

Just noticed the maximize/unmaximize animation is very ugly due to this bug. If you maximize the System Settings window for example, it judders and wobbles during the animation.

[Daniel van Vugt (vanvugt)]

BTW I've just found out that Windows 10 seems to implement both approaches to handling resize lag, depending on which app you look at (!?)

Legacy apps with the old look:
Border gaps at the bottom/right filled with white.

Newish apps:
Texture clamping is used so border gaps are filled with the edge pixel colour. You can sometimes see the edge pixels (only) get stretched.

[Michal Predotka (mpredotka)]

I was about to report a new bug about resizing but I found this one.
I've noticed the problem is only with right and bottom border of an app window. the top and left are fine whenever side I start resizing from. Also it looks like the title bar does not stutter when resizing, just the rest of the window. I've recorded a short video about it: https://youtu.be/3H48UBpXUUo

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Daniel d'Andrada (dandrader)]

@mpredotka

This looks more like bug 1540702.

[Michal Predotka (mpredotka)]

@dandrader thanks, will add my comment there too.

[Alberto Salvia Novella (es20490446e)]

@ Michał Sawicz

When overriding my settings, please provide an explanation about why you are doing so.

[Daniel d'Andrada (dandrader)]

I suppose it's because the people who work on those projects (and few other stakeholders) are the ones who decide the importance (hence *priority*) of bugs in them.

Example: critical bugs are worked on ASAP. But if you made this bug critical yourself, for instance, it doesn't mean the team sees it as critical and will therefore start working on it right away. If this bug was then left as critical even though the team didn't sees it as such, the Importance field would lose its meaning and usefulness.

[Alberto Salvia Novella (es20490446e)]

@ Daniel d'Andrada

What importance Ubuntu packages should have is determined at (https://wiki.ubuntu.com/Bugs/Importance). And can only be set by members of the Ubuntu Bug Control team, who had proved beforehand they know how to do it.

In my case I have been setting importance for bugs in Ubuntu nearly for three years now, and I have done it for thousands of bugs. In fact I train people in Ubuntu on how to do it.

For my workflow is important that all bugs have priority set. So if you see a priority is wrong feel free to change it commenting about the reasons. But leave it differently than "undecided", as this signals I should set the importance.

[Daniel d'Andrada (dandrader)]

In which case I say it's medium as it doesn't impede any use case. It just makes Unity 8 looks better (as in eye candy, smoother animations etc).

[Daniel van Vugt (vanvugt)]

I think it's dangerous to use the term "eye candy" here. This is about professionalism. If Unity8 doesn't feel very professional, people won't want to use it.