[regression] [mako] Scrolling and animations are very slow with Mir

[Omer Akram]

Image 80

A few weeks ago when I was running Unity8 on Mir everything was pretty fast and responsive but a few days ago rendering got much slower. Normal animations are laggy, scrolling in dash is laggy/jerky. App focus animation is slow. And if you really look close at a spinner even the spinner in the bluetooth panel of the system-settings app its rotation is even not smooth.

So basically all type of rendering is slow.

Saviq says if you just run autopilot test suite of ubuntu_calculator_app (which opens the quite a few times) you will start noting the slowness very easily after the testsuite completes.

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: libmirplatform 0.0.13+13.10.20131003-0ubuntu1
Uname: Linux 3.4.0-3-mako armv7l
ApportVersion: 2.12.5-0ubuntu1
Architecture: armhf
Date: Fri Oct 4 11:10:33 2013
InstallationDate: Installed on 2013-10-03 (0 days ago)
InstallationMedia: Ubuntu Saucy Salamander (development branch) - armhf (20131003.2)
MarkForUpload: True
ProcEnviron:
 TERM=linux
 PATH=(custom, no user)
SourcePackage: mir
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michał Sawicz (saviq)]

I checked top, iotop, free - CPU was idle, IO was idle, no swap was used - the device was generally idle, but unity8 is slow as hell.

Steps to reproduce: for me it happens when running an autopilot suite, but it's probably equivalent to starting and stopping apps.

[Gema Gomez (gema)]

In my case, I left the phone running mir overnight and in the morning it was so slow it was unusable. Top didn't seem to indicate there is any process eating up all the CPU.

[Fabian Herb (fherb)]

Maybe it's related: Unity eats up a lot of CPU in the latest version, but not on mir but surfaceflinger, resulting in very bad responsiveness. Sometimes it's unusable, and "sudo reboot" doesn't work anymore...

top snapshot while doing nothing:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1614 phablet 20 0 263m 77m 47m S 62.3 4.1 2:13.67 unity8
  534 system 20 0 70304 42m 39m S 12.1 2.2 1:12.00 surfaceflinger
 2083 phablet 30 10 310m 126m 37m R 7.4 6.8 1:21.44 QtWebProcess
 3691 phablet 20 0 4540 1256 784 R 3.5 0.1 0:00.61 top
 3501 root 20 0 0 0 0 S 3.2 0.0 0:03.79 kworker/u:33
 3503 root 20 0 0 0 0 S 2.4 0.0 0:01.29 kworker/u:35
...

Version: 7.82+13.10.20131004.1-0ubuntu1

[Kevin DuBois (kdub)]

any complaints in logcat or dmesg?

[Gerry Boland (gerboland)]

Here's result of profiling: http://pastebin.ubuntu.com/6192831/
Looks to be graphical, needs Mir team help.

[Gerry Boland (gerboland)]

I've also noticed on GalaxyNexus there's some idling CPU usage (about 5%) from a httpThread - which is one of Qt's internal thread for http downloads - likely culprit is a lens having shell fetch images. Needs further investigation

[Omer Akram (om26er)]

I think we need to flash olders images on a phone and see where the performance drop happened. I think the slowness goes back atleast 2 weeks and its only noted now because we (the testers) started using Mir enabled on our daily driver phone just recently.

[Omer Akram (om26er)]

(I would do that but I have a crappy internet :/)

[Daniel van Vugt (vanvugt)]

We reserve "Critical" for bugs which make the system completely unusable. This doesn't quite qualify as Critical, but probably High.

[Omer Akram (om26er)]

I think we are using the bug importance wrong then. When the problem is
visible enough that its going to affect the usage of the phone every single
second in a bad way. Its more than critical.

On Mon, Oct 7, 2013 at 12:04 PM, Daniel van Vugt <
<email address hidden>> wrote:

> We reserve "Critical" for bugs which make the system completely
> unusable. This doesn't quite qualify as Critical, but probably High.
>
> ** Changed in: mir
> Importance: Critical => High
>
> ** Changed in: unity8
> Importance: Critical => High
>
> ** Changed in: mir (Ubuntu Saucy)
> Importance: Critical => High
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1235190
>
> Title:
> [mako] Unity8 on Mir got slow
>
> Status in Mir:
> Confirmed
> Status in The Unity 8 shell:
> Confirmed
> Status in “mir” package in Ubuntu:
> Confirmed
> Status in “mir” source package in Saucy:
> Confirmed
>
> Bug description:
> Image 80
>
> A few weeks ago when I was running Unity8 on Mir everything was pretty
> fast and responsive but a few days ago rendering got much slower.
> Normal animations are laggy, scrolling in dash is laggy/jerky. App
> focus animation is slow. And if you really look close at a spinner
> even the spinner in the bluetooth panel of the system-settings app its
> rotation is even not smooth.
>
> So basically all type of rendering is slow.
>
> Saviq says if you just run autopilot test suite of
> ubuntu_calculator_app (which opens the quite a few times) you will
> start noting the slowness very easily after the testsuite completes.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 13.10
> Package: libmirplatform 0.0.13+13.10.20131003-0ubuntu1
> Uname: Linux 3.4.0-3-mako armv7l
> ApportVersion: 2.12.5-0ubuntu1
> Architecture: armhf
> Date: Fri Oct 4 11:10:33 2013
> InstallationDate: Installed on 2013-10-03 (0 days ago)
> InstallationMedia: Ubuntu Saucy Salamander (development branch) - armhf
> (20131003.2)
> MarkForUpload: True
> ProcEnviron:
> TERM=linux
> PATH=(custom, no user)
> SourcePackage: mir
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mir/+bug/1235190/+subscriptions
>

[Loïc Minier (lool)]

There are various causes of slowdowns; Mir is slightly slower than SurfaceFlinger on rendering, but that's expected and performance optimizations are upcoming (post-13.10). One known reason for slow downs is when crash files are being written; typically if a large process crashes, the phone is stuck for a while writing the .crash file.

Lastly, there's a known memory hog in upstart using all memory very rapidly on Mir builds, notably mako; this is being worked on.

[Kevin DuBois (kdub)]

I'm a bit concerned that the clocks of the gpu or the memory bus are going low inappropriately. Still haven't been able to reproduce, but that'll be the first thing I check

[Kevin DuBois (kdub)]

upgrading my previous statement from 'concerned' to 'strongly suspect'

[Thomi Richards (thomir-deactivatedaccount)]

I just reproduced this on image 84. I'm running unity8 on mir. All I did to reproduce it is run the file-manager autopiot test suite several times, which makes me think it has something to do with launching and closing applications.

See attached video...

[Kevin DuBois (kdub)]

I see wild (and explainable) mood swings in mir performance when we have our clocks kicked out from under us. When this problem occurs, try turning clocks up to 11:

echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

This at least turns the cpu clocks to high and probably turns the axi bus high.
i'd expect this to improve framerates from about 15->vsync (or at least that's what i see in my mir demo programs). If this is the case we're fighting over clocks with powerd.

[Kevin DuBois (kdub)]

my current working theory is that powerd (when it talked to surfaceflinger) would just hang when we running mir, because we would forcibly stop the surfaceflinger. Now that powerd talks to unity, which talks to mir, so it is starting to make actual power requests, including scaling back the clocks that mir needs to operate at full speed.

[Omer Akram (om26er)]

I can verify that command brings back the performance to a very reasonable level.

[Omer Akram (om26er)]

Also your theory makes it sounds like we are hit by bug 1233257. On SF as you said powerd would just hang so we don't see the performance issue there once the screen is unlocked.

[kevin gunn (kgunn72)]

ok - this may require a 2 phased approach to "fixing"
first, for the quick but less-than-stellar battery performance fix would be to force the pm policy to "performance"
second, would be to find the correct PM constraint to call into (a most likely qcomm specific kernel pm fmwk) while rendering & then remove the constraint when rendering is complete

[Oliver Grawert (ogra)]

i really dont think this is anyway nearly the right approach but here would be a debdiff to force performance
http://paste.ubuntu.com/6209409/

nobody in this bug collected any data but using their gut feelings apparently though, can we at least have some top output (separately for CPU and RAM please) while this incident happens, as well, as dmesg output and i.e. powerd logs etc ...

[Oliver Grawert (ogra)]

(cranking up the CPU wont help at all on the GPU in an arm system)

[Michał Sawicz (saviq)]

I for one have mentioned at the beginning that CPU wasn't a problem here. CPU was idle, RAM was aplenty, swap was empty.

[Kevin DuBois (kdub)]

its not the cpu itself, when you force the cpu to performance, you also force the system bus to stay on.
So you can have a low percentage of ram, low cpu usage, and still be going slow because the pixels aren't making it back and forth from the gpu to the ram to the screen fast enough. the test was primarily about the system bus, not the cpu

[Kevin DuBois (kdub)]

and just to be clear (seems to have been some confusion about this) turning the cpu to 'performance' is not a landable solution because of battery life. It is a good test to see if the problem is the clocks are turned to low (a system power management problem), or mir itself is getting stucks somewhere (a problem isolated to mir code)

[Jim Hodapp (jhodapp)]

I see a very similar slowdown when playing a video in mediaplayer-app, quit the app, play again, and a couple of more cycles and all rendering in the shell and other places all become painfully slow. Eventually the mediaplayer-app itself hangs because it can't render the video anymore.

[Omer Akram (om26er)]

Updated the title. Also to point out that this is not similar to bug 1182930 as the performance on Mako has been pretty good in the past. I traced back the problem between image 63-65. I tested image 62 and everything is smooth. image 63 and 64 don't exist (got deleted in system-image.ubuntu.com) so didn't test them so image 65 is the one where I first noted the issue.

[Omer Akram (om26er)]

You can flash image 62 by: phablet-flash ubuntu-system --channel devel --revision 62 --no-backup

[Jim Hodapp (jhodapp)]

Moved the issue I'm seeing with video to: https://bugs.launchpad.net/ubuntu/+source/mir/+bug/1237586

[Kevin DuBois (kdub)]

trying to keep this bug focused :)

lool's comment #13, noting a memory leak in upstart, would only affect longer term mir usage... since we see the bug very quickly, this is probably not what we're seeing

comments 14 15 17 18 20 21 22 23 24 25 26 all have to do with https://bugs.launchpad.net/powerd/+bug/1233257
although this causes a problem, resolving this does not affect the apparent jumping in the animations, so those can be discarded

comment 27 is being filed as a separate bug to avoid muddying this bug any further

Currently, the easiest way to reproduce this bug currently is to flash image 89 and scroll in the applications lens.

[Kevin DuBois (kdub)]

ogra is spot on in https://bugs.launchpad.net/ubuntu/+source/mir/+bug/1235190/comments/22 about having to collect more quantifiable data about what is reported as 'slowness'

we have some profile data collected from rsalveti that indicates the cpu is more than 50% idle when this is happening: http://paste.ubuntu.com/6214763/

[Kevin DuBois (kdub)]

between image 62/65, we added more waits that ensure the gpu is done rendering (this was a bug in mir that caused flickering on screen). However, this did introduce more waiting during the rendering pass, and I think it pushed us from 16ms frames (60fps) to a bit over (perhaps 19-20ms). This causes us to miss some vsyncs.

Mir uses these wait fences naively, surfaceflinger is a bit smarter in passing them around. We had plans in the works to pass these fences around (and reduce waiting)... let me see if i can think of a more short-term solution.

[Daniel van Vugt (vanvugt)]

Linking Kevin's branch which *sounds* related.

[Kevin DuBois (kdub)]

yes, linked branch improves performance by shifting where we wait for render completion out of the unity8 render thread and into mir's composition pass. This better pipelines the composition pass with the render pass, and gets performance to vsync rate. Performance improvement was verified by ricmm, tvoss, racarr, kgunn.

Something in that branch currently causes a screenshot to appear white during app launch, am trying to fix before landing.

[Daniel van Vugt (vanvugt)]

I don't think this bug is mako-specific any more. It's a very visible regression on nexus7/grouper too. Can we remove mentions of "mako"?

[Omer Akram (om26er)]

I think Mir was never working Nexus 7 for us to verify if the performance
regressed there as well. In case of mako things were pretty smooth and
fast(under Mir) but later regressed.

On Mon, Oct 14, 2013 at 12:14 PM, Daniel van Vugt <
<email address hidden>> wrote:

> I don't think this bug is mako-specific any more. It's a very visible
> regression on nexus7/grouper too. Can we remove mentions of "mako"?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1235190
>
> Title:
> [regression] [mako] Scrolling and animations are very slow with Mir
>
> Status in Mir:
> In Progress
> Status in The Unity 8 shell:
> Confirmed
> Status in “mir” package in Ubuntu:
> Confirmed
> Status in “mir” source package in Saucy:
> Confirmed
>
> Bug description:
> Image 80
>
> A few weeks ago when I was running Unity8 on Mir everything was pretty
> fast and responsive but a few days ago rendering got much slower.
> Normal animations are laggy, scrolling in dash is laggy/jerky. App
> focus animation is slow. And if you really look close at a spinner
> even the spinner in the bluetooth panel of the system-settings app its
> rotation is even not smooth.
>
> So basically all type of rendering is slow.
>
> Saviq says if you just run autopilot test suite of
> ubuntu_calculator_app (which opens the quite a few times) you will
> start noting the slowness very easily after the testsuite completes.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 13.10
> Package: libmirplatform 0.0.13+13.10.20131003-0ubuntu1
> Uname: Linux 3.4.0-3-mako armv7l
> ApportVersion: 2.12.5-0ubuntu1
> Architecture: armhf
> Date: Fri Oct 4 11:10:33 2013
> InstallationDate: Installed on 2013-10-03 (0 days ago)
> InstallationMedia: Ubuntu Saucy Salamander (development branch) - armhf
> (20131003.2)
> MarkForUpload: True
> ProcEnviron:
> TERM=linux
> PATH=(custom, no user)
> SourcePackage: mir
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mir/+bug/1235190/+subscriptions
>

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:~mir-team/mir/development-branch at revision None, scheduled for release in mir, milestone Unknown

[Daniel van Vugt (vanvugt)]

Fix committed to lp:mir at revision 1100.

[Timo Jyrinki (timo-jyrinki)]

Fix released in https://launchpad.net/ubuntu/+source/mir/0.0.15+13.10.20131014-0ubuntu1 (changelog at https://code.launchpad.net/~mir-team/mir/development-branch/+merge/191055).

[Daniel van Vugt (vanvugt)]

Another significant performance boost is coming in Mir 0.1.0.

And after that, some time, we hope to finish this:
https://bugs.launchpad.net/mir/+bug/1227739/comments/10