Ubuntu Firefox is 15% slower than Flatpak Firefox for speedometer benchmark

[Andrei Petcu]

Steps to reproduce:

I installed Firefox Flatpak (https://flathub.org/apps/details/org.mozilla.firefox) and Chrome.

I executed closed all apps and processes and one by one I executed Speedometer test (https://browserbench.org/Speedometer2.0/) in each browser 3 times. I took the max score for reach browser out of the 3 executions of the test.

Actual results:

Results
100.3 - Firefox Flatpak
85.42 - FIrefox DEB (pre-installd from the store)
100.3 - Google Chrome

The preinstalled DEB package Firefox was slower than both Firefox Flatpak and Chrome

Expected results:

I expected Firefox DEB and Firefox Flatpak to have the same results.

[Olivier Tilloy (osomon)]

Could you please run `apport-collect 1908082` to attach relevant debug information to this bug? Thanks!

[Vincent Chernin (vchernin)]

I can confirm this issue using the given steps on pop os 20.10. All tests were run on the latest firefox and chrome versions (84 and 87, respectively) as of 2020-12-24. New profiles/users were used for each test (via about:profiles in firefox and the user feature in chrome). All other apps were closed for the tests.

76.3 - Firefox (Deb) (from ubuntu 20.10)
87.5 - Firefox (Flatpak)
86.6 - Firefox (Snap)
85.4 - Google Chrome (Deb)

Maybe there is a compilation difference between ubuntu's firefox and other firefox sources.

[Andrei Petcu (andreipetcu)]

Sorry, I switched distros. I'm on Fedora now. I see Vincent reproduced it. Can you please collect the bug info Vincent?

[Sebastien Bacher (seb128)]

Could you give details on the firefox versions used in each of the cases?

[Vincent Chernin (vchernin)]

Sorry this is a bit late, but here once again are the speedometer benchmarks. This time I made sure to include the package version.

I used a different computer than my first test but the conclusion is still the same: Firefox deb is beaten by alternative packages of firefox (in this case snap).

Speedometer score (https://browserbench.org/Speedometer2.0/):
Tested on Ubuntu 20.04.2 (ISO: ubuntu-20.04.2.0-desktop-amd64.iso)

Firefox Snap:
Score: 36.6±2.0
Package: 86.0-3

Firefox Deb:
Score: 28.89±0.18
Package: 86.0+build3-0ubuntu0.20.04.1

For further research it may be useful to benchmark firefox deb vs firefox snap/flatpak in various tests, similarly to how phoronix compares chrome and firefox on linux (https://www.phoronix.com/scan.php?page=news_item&px=Chrome-89-Firefox-86).

However, even though the speedometer score is evidently lower on the deb that doesn't mean it is necessarily an real-world issue. As noted by google when they retired octane as a recommended benchmark (tests javascript), web benchmarks can be misleading and can poorly represent real-world performance (https://v8.dev/blog/retiring-octane). The lower score on firefox deb may not correlated with a perceiveable performance difference.

My opinion is that a fix for this issue is not worth putting considerable time into if we cannot demonstrate clear real-world performance deficiencies in firefox deb.

[Andrei Petcu (andreipetcu)]

There are 2 real world benchmarks all browsers care about: Page load and Speedometer. Octane and Speedometer are polar opposites. https://blog.chromium.org/2017/04/real-world-javascript-performance.html?m=1

I'm certain that this 15% performance dip for deb is visible in real world.

[Olivier Tilloy (osomon)]

Similarly, I'm seeing better performance for the snap version of firefox than with the deb, on Ubuntu 20.10:

  deb: 85.1 ± 1.4
  snap: 108 ± 1.5

This could very well be explained by the toolchain and default build options used to build each package: the snap is built from an upstream binary, compiled with a very recent toolchain and likely many optimizations turned on, whereas the deb uses the toolchain available in the corresponding Ubuntu series, and often not with full optimizations to work around hardware limitations in the Launchpad build farm or toolchain bugs.

This is not to say that we cannot research turning on more optimizations though.

[Tynach (tynach2)]

@osomon, I don't think this is due to the toolchain version used. I semi-recently (I don't know how long it's been going on.. Somewhere in the span of the last 6 months to the last 2 years? Probably more like the past year and a half, but it feels in some ways longer, in some ways shorter?) found that none of the shaders I've written on Shadertoy can run at 60fps. Not even relatively simple ones like this one:
https://www.shadertoy.com/view/MtGcRW

There's no reason for this to run at 40 - 55 fps, but it does. In fact, no matter how simple I make a shader - even just showing a static unchanging color - Firefox will never run it at 60fps.

This might have started when I upgraded to 20.04 from 18.04, but I'm unsure.. I feel like I'd have remembered when it started better if it was definitively after a major upgrade (I use Shadertoy frequently). I do know I've tried using a blank profile, and also tried disabling all add-ons and so on. I've tried using WebRender, having WebGL run in a separate process, and all sorts of other tweaks via about:config. Nothing gets any Shadertoy shaders to 60fps.

However, they all run at 60fps easily in the version downloaded from Mozilla's website. What's more, is that I've been using Firefox instead of Chrome for the past few years, and when I first switched over the Ubuntu packages worked fine and ran shaders at full speed.

I remember that when I first noticed Firefox not running them at full speed, I assumed it was Shadertoy's fault. Maybe they'd left some debugging code enabled in the Javascript, and that was slowing it down. So I just figured I'd ignore it and wait for it to be fixed by them.

After a month or two of waiting, and seeing the site actually update with some minor changes here and there without it being fixed, I decided to test Chrome. Ran fine. Tried all sorts of tweaks, nothing helped Firefox run it faster. I still thought it was Shadertoy's code that changed and was just not optimized for Firefox. Then a few months after the problem started, I finally tried running a copy of Firefox from Mozilla... And it ran just fine.

I don't know if that's the same problem as here, but this bug was reported within the window of time when I remembered seeing the slowdown (granted, that's a huge window due to my poor memory). And if it is, then it can't possibly be only because of being compiled on an old version of the compiler toolchain, as it used to run faster than it does now.

[Olivier Tilloy (osomon)]

@Tynach, the symptoms you describe do advocate in favour of the toolchain/optimizations hypothesis. If upstream builds are not affected, wherease Ubuntu builds are, when both are run on the same platform, surely there is a difference in the optimizations used to produce the binary under test.

That said I've tested your sample shader in shadertoy on Ubuntu 21.04, and while the framerate is not super constant it does hit 60FPS regularly (chromium is behaving better, also around 60FPS but with a much more constant framerate than firefox). That's with the firefox Ubuntu build, and the chromium snap.

[Vincent Chernin (vchernin)]

I found this article from Martin Stransky: https://mastransky.wordpress.com/2019/01/08/fedora-firefox-heads-to-updates-with-pgo-lto/

It appears Fedora's Firefox package is build with PGO. Is Ubuntu's .DEB package built with PGO as well?

There's also this documentation from Mozilla: https://firefox-source-docs.mozilla.org/build/buildsystem/pgo.html

[Tynach (tynach2)]

@Olivier Tilloy, ironically, I'm now seeing the same behavior in version 90. I don't remember if the performance behaved the same way in 89 or not, but maybe? At any rate, performance of Ubuntu's builds has vastly improved, even if it's still not quite on the same level as Chrome. I haven't re-tested non-Ubuntu Firefox yet either.

Also, I'm on 20.04, which might be making a difference.

[Launchpad Janitor (janitor)]

[Expired for firefox (Ubuntu) because there has been no activity for 60 days.]

[Tynach (tynach2)]

60 days is an absurd timeout, and I would complain loudly about that...

... If it weren't for the fact that I came back to this bug report to say that it seems to have very recently been solved. Had to have been in version 93 or 94, probably 94.

At least, for my own purposes, it seems completely solved. I'm looking at Shadertoy shaders, and getting a solid 60fps instead of wavering between 50 and 57. I look at these frequently, and feel like I would have noticed sooner than now if the fix happened any sooner than version 94 (or maayybe 93).

Can't say whether or not OP's problem is solved. I haven't done any rigorous benchmarking either, so it might just be Shadertoy improving their framerate detection code. I just know I suddenly no longer need to sit there for several minutes before seeing a framerate of 59 reported, for a single second, before going 57 or lower again the rest of the time.