Please compile Firefox with PGO optimizations

[Sorin Ionescu]

Binary package hint: xulrunner-1.9

I've been using Arch Linux for the last 3 months. Now, I'm back on Ubuntu because I got tired of configuring. Configuring never ends. However, for Arch Linux, I made the best Firefox 3 package ( http://aur.archlinux.org/packages.php?ID=15184 ), which includes PGO optimizations ( http://developer.mozilla.org/en/docs/Building_with_Profile-Guided_Optimization ).

The Windows version is PGO optimised. People will hate the Ubuntu version knowing that the Windows version is so much faster. The official mozilla nightlies are PGO optimised.

This was a test I did on Arch Linux, which is compiled against i686:
Firefox 3 Beta 3 on the left.
Firefox 3 Beta 4 with Profile Guided Optimisations on the right: http://www.paste2.org/p/15666

This is what I got with this build https://bugs.launchpad.net/gutsy-backports/+bug/212468

http://tinyurl.com/6fv8aq

7 times slower. I'm not sure how much i686 helps. But, my i686 PGO was 3 times faster than my normal i686.

There are no regressions. It just works.

I've attached how it was compiled and the mozconfig.

[Ted Mielczarek (ted-mielczarek)]

This was actually ok before bug 361343. There was another older bug where I made the existing PGO support work again.

[Ted Mielczarek (ted-mielczarek)]

Created an attachment (id=304838)
enable pgo on fx-linux-tbox

There you go.

[Ted Mielczarek (ted-mielczarek)]

Created an attachment (id=304854)
with clobbery

So, this is going to be an unpopular change, I think we need to make it clobber as well.

[Joduinn (joduinn)]

From offline discussions with Damon, RobSayre, and then Ted:

1) This is being proposed for nightly/clobber builds, as well as hourly/incremental builds as well as release builds.

2) There are similar changes required for Mac. Ted is filing a bug for that.

3) This double-compile process will take extra time to cycle a build; seems like linux would be 2x, win32 would be just under 2x. This will need to be communicated to everyone watching Tinderbox.

4) We are not blocked on bug#418896 or bug#418894.

[Joduinn (joduinn)]

marking P1 as its needed for beta4.

[RobertSayre (sayrer)]

I had to back this out because of the following error:

/builds/tinderbox/Fx-Trunk/Linux_2.6.18-8.el5_Depend/mozilla/memory/jemalloc/jemalloc.c: In function 'choose_arena':
/builds/tinderbox/Fx-Trunk/Linux_2.6.18-8.el5_Depend/mozilla/memory/jemalloc/jemalloc.c:6043: error: corrupted profile info: edge from 5 to 6 exceeds maximal count
gmake[4]: *** [jemalloc.o] Error 1

[RobertSayre (sayrer)]

Looks like this might be just jemalloc causing problems. We should be able to skip PGO for that.

[Ted Mielczarek (ted-mielczarek)]

Created an attachment (id=305385)
skip PGO in jemalloc

This should do it.

[Benjamin Smedberg (Mozilla) [:bs] (benjamin-smedbergs)]

(From update of attachment 305385)
File a followup? I imagine PGO for jemalloc would useful if simply for branch-prediction/locality optimizations.

[Ted Mielczarek (ted-mielczarek)]

(From update of attachment 305385)
Filed bug 419470, will mention it in a comment.

[Dsicore (dsicore)]

(From update of attachment 305385)
a1.9+=damons

[Ted Mielczarek (ted-mielczarek)]

(From update of attachment 305385)
sayrer: you should be clear to try this again, but given that we're freezing tomorrow night, it might be rough to do it before wednesday.

[Beltzner (beltzner)]

Do you guys need a clean window to do this, like with Windows?

[RobertSayre (sayrer)]

(In reply to comment #13)
> Do you guys need a clean window to do this, like with Windows?

That would be best, but I don't think it's critical we do it this week. Our best avenue for beta linux users seems to be distro betas. So I'm content to focus on mac and additional Windows profile input for the rest of this week.

[Beltzner (beltzner)]

(From update of attachment 304854)
a=beltzner for after beta 4

[Joduinn (joduinn)]

*** Bug 384899 has been marked as a duplicate of this bug. ***

[Joduinn (joduinn)]

(In reply to comment #15)
> (From update of attachment 304854 [details])
> a=beltzner for after beta 4

Therefore, removing as blocker on bug#418926 (3.0beta4 tracking bug)

[Ted Mielczarek (ted-mielczarek)]

Rob: do you want to do this? This has been sort of hanging out here for a while.

[Mtschrep (mtschrep)]

Given limited time in the release we are punting pgo on mac and linux - let's pick this up in next release

[Sorin Ionescu (sorin.ionescu-deactivatedaccount)]

Binary package hint: xulrunner-1.9

I've been using Arch Linux for the last 3 months. Now, I'm back on Ubuntu because I got tired of configuring. Configuring never ends. However, for Arch Linux, I made the best Firefox 3 package ( http://aur.archlinux.org/packages.php?ID=15184 ), which includes PGO optimizations ( http://developer.mozilla.org/en/docs/Building_with_Profile-Guided_Optimization ).

The Windows version is PGO optimised. People will hate the Ubuntu version knowing that the Windows version is so much faster. The official mozilla nightlies are PGO optimised.

This was a test I did on Arch Linux, which is compiled against i686:
Firefox 3 Beta 3 on the left.
Firefox 3 Beta 4 with Profile Guided Optimisations on the right: http://www.paste2.org/p/15666

This is what I got with this build https://bugs.launchpad.net/gutsy-backports/+bug/212468

http://tinyurl.com/6fv8aq

7 times slower. I'm not sure how much i686 helps. But, my i686 PGO was 3 times faster than my normal i686.

There are no regressions. It just works.

[Ted Mielczarek (ted-mielczarek)]

(From update of attachment 304854)
Clearing approval flag to get this off the radar.

[Ted Mielczarek (ted-mielczarek)]

(From update of attachment 305385)
Clearing approval flag to get this off the radar.

[Ted Mielczarek (ted-mielczarek)]

We may take this in 3.next.

[Simon Strandman (nejsimon)]

Building firefox with PGO is actually quite simple. Just add "mk_add_options PROFILE_GEN_SCRIPT='$(PYTHON) $(MOZ_OBJDIR)/_profile/pgo/profileserver.py'" to mozconfig (profileserver.py is a script that runs the app) and run make with the "profiledbuild" target instead of "build".

Please enable this in Jaunty!

[John Vivirito (gnomefreak)]

nxsty,
Please feel free to build it with PGO and if it works after testing it please feel free to offer a debdiff or join us in #ubuntu-mozillateam on irc.freenode.net and talk to one of us about it and we can see if it is agreed opon

[Ted Mielczarek (ted-mielczarek)]

I wouldn't block on it, though.

[Bzbarsky (bzbarsky)]

So I just tried compiling with pgo on Linux. I get:

../../../../db/morkreader/libmorkreader_s.a(nsMorkReader.o): In function `global constructors keyed to 65535_0_nsMorkReader.cpp':
/home/bzbarsky/mozilla/debug/mozilla/db/morkreader/nsMorkReader.cpp:579: undefined reference to `__gcov_init'
../../../../db/morkreader/libmorkreader_s.a(nsMorkReader.o):(.data.rel+0x24): undefined reference to `__gcov_merge_add'
../../../../db/morkreader/libmorkreader_s.a(nsMorkReader.o):(.data.rel+0x30): undefined reference to `__gcov_merge_single'
../../../../db/morkreader/libmorkreader_s.a(nsMorkReader.o):(.data.rel+0x3c): undefined reference to `__gcov_merge_single'
../../../../db/morkreader/libmorkreader_s.a(nsMorkReader.o):(.data.rel+0x48): undefined reference to `__gcov_merge_add'
../../../../db/morkreader/libmorkreader_s.a(nsMorkReader.o):(.data.rel+0x54): undefined reference to `__gcov_merge_ior'
/home/bzbarsky/mozilla/debug/obj-firefox-gpo/dist/lib/libunicharutil_s.a(nsUnicharUtils.o): In function `global constructors keyed to 65535_0_nsUnicharUtils.cpp':
/home/bzbarsky/mozilla/debug/obj-firefox-gpo/intl/unicharutil/util/internal/nsUnicharUtils.cpp:226: undefined reference to `__gcov_init'
/home/bzbarsky/mozilla/debug/obj-firefox-gpo/dist/lib/libunicharutil_s.a(nsUnicharUtils.o):(.data.rel+0x24): undefined reference to `__gcov_merge_add'
/home/bzbarsky/mozilla/debug/obj-firefox-gpo/dist/lib/libunicharutil_s.a(nsUnicharUtils.o):(.data.rel+0x30): undefined reference to `__gcov_merge_single'
collect2: ld returned 1 exit status
gmake[7]: *** [libplaces.so] Error 1

[Reed Loden (reed)]

I think Murali was using gcov on Linux lately... maybe he can give some pointers?

[Murali Nandigama (mnandigama)]

Preliminary symptoms seems to be issues with LDFLAGS.

I need to check the .mozconfig file to comment further.

[Reed Loden (reed)]

bz, please post your .mozconfig for Murali, as per comment #26.

[Bzbarsky (bzbarsky)]

Created an attachment (id=362727)
The mozconfig in question

I had to do some manual copy/paste inclusion (I assumed you didn't want me to attach the 4-5 mozconfig files involved that include each other), and I removed all the comment lines. Other than that, this is the mozconfig involved.

Building on FC9 with:

% g++ --version
g++ (GCC) 4.3.0 20080428 (Red Hat 4.3.0-8)
% ld --version
GNU ld version 2.18.50.0.6-7.fc9 20080403

and ccache enabled.

[Murali Nandigama (mnandigama)]

Please add the following line to your .mozconfig file

export LDFLAGS="-lgcov -static-libgcc"

However, on a separate note, you are using '-g' flag for CFLAGS and CXXFLAGS and also using 'disable-debug'. Is it by design.

Thanks
Murali

[Ted Mielczarek (ted-mielczarek)]

Last time I tried this (with an older GCC, definitely not 4.3) it didn't require any special mozconfig hacking. It's possible gcc just sucks.

[Bzbarsky (bzbarsky)]

> export LDFLAGS="-lgcov -static-libgcc"

I can try that, but sounds like https://developer.mozilla.org/en/Building_with_Profile-Guided_Optimization needs updating. Or the build target should perhaps handle this automatically.

AS for -g and --disable-debug, yes. I want to be able to have debug symbols in my optimized builds, both for crash stacks and actual debugging of issues that only show up in the optimized build.

[L. David Baron (dbaron)]

Reed, why do you think gcov is the same as PGO? They're very different things, although I could understand if they shared some common infrastructure. Is there something I'm missing?

[Ted Mielczarek (ted-mielczarek)]

(In reply to comment #31)
> > export LDFLAGS="-lgcov -static-libgcc"
>
> I can try that, but sounds like
> https://developer.mozilla.org/en/Building_with_Profile-Guided_Optimization
> needs updating. Or the build target should perhaps handle this automatically.

The build system should handle this all automatically. It's not doing anything complicated, just -fprofile-generate / -fprofile-use:
http://mxr.mozilla.org/mozilla-central/source/configure.in#7030

[Reed Loden (reed)]

(In reply to comment #32)
> Reed, why do you think gcov is the same as PGO? They're very different things,
> although I could understand if they shared some common infrastructure. Is
> there something I'm missing?

I only mentioned gcov because of what the errors in comment #24 said.

[Bzbarsky (bzbarsky)]

Adding those LDFLAGS doesn't help; I still get the error from comment 24. The relevant command running at the time is:

c++ -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-long-long -pedantic -g -fno-strict-aliasing -fshort-wchar -pthread -pipe -DNDEBUG -DTRIMMED -g -fno-inline -Os -freorder-blocks -fno-reorder-functions -fPIC -shared -Wl,-z,defs -Wl,-h,libplaces.so -o libplaces.so nsAnnoProtocolHandler.o nsAnnotationService.o nsFaviconService.o nsNavHistory.o nsNavHistoryExpire.o nsNavHistoryQuery.o nsNavHistoryResult.o nsNavBookmarks.o nsMaybeWeakPtr.o nsMorkHistoryImporter.o nsPlacesModule.o nsNavHistoryAutoComplete.o -lpthread -lgcov -static-libgcc -Wl,-rpath-link,/home/bzbarsky/mozilla/debug/obj-firefox-pgo/dist/bin -Wl,-rpath-link,/usr/local/lib ../../../../db/morkreader/libmorkreader_s.a /home/bzbarsky/mozilla/debug/obj-firefox-pgo/dist/lib/libunicharutil_s.a -L/home/bzbarsky/mozilla/debug/obj-firefox-pgo/dist/bin -lxpcom -lxpcom_core -L/home/bzbarsky/mozilla/debug/obj-firefox-pgo/dist/bin -L/home/bzbarsky/mozilla/debug/obj-firefox-pgo/dist/lib -lplds4 -lplc4 -lnspr4 -lpthread -ldl -Wl,--version-script -Wl,/home/bzbarsky/mozilla/debug/mozilla/build/unix/gnu-ld-scripts/components-version-script -Wl,-Bsymbolic -lasound -ldl -lm

[John Vivirito (gnomefreak)]

This will not land in 3.0 since it is stable release and should only get security updates, however i think we should work on it for 3.1 and up.

[Tglek (tglek)]

(In reply to comment #70)
> Taras, thankyou for the information. I tried building firefox using mozilla's
> pgo instructions, but whenever I try building it, I get the following error
> message mid-way through the build:
>
> OBJDIR=/home/richard/firefox-build/vanilla/mozilla-1.9.2/../firefox-build
> python
> /home/richard/firefox-build/vanilla/mozilla-1.9.2/../firefox-build/_profile/pgo/profileserver.py
> INFO | automation.py | Application pid: 28455
> TEST-UNEXPECTED-FAIL | automation.py | Exited with code -11 during test run
> INFO | automation.py | Application ran for: 0:00:00.109925
> make: *** [profiledbuild] Error 245

I am guessing that this this due to the same segfault as others have reported(on 64bit). This may be due to our buildsystem occasionally omitting pgo flags. The other approach to doing pgo is to build with
export PGO="-fprofile-generate=/tmp/pgo -fprofile-arc"
export CXX="g++ $PGO"
export CC="cc $PGO"
in your .mozconfig
then run firefox, then build it a second time with PGO=-fprofile-use=/tmp/pgo

It would be cool if someone could figure out how to build firefox with pgo on 64bit.

[Mh+mozilla (mh+mozilla)]

I don't see why the flags would be sometimes ommitted when building on x86-64 and not on x86. My guess is that the problem lies in gcc. Even building with CXX="g++ --coverage" and CC="gcc --coverage" results in the same crashes.

[Tglek (tglek)]

(In reply to comment #72)
> I don't see why the flags would be sometimes ommitted when building on x86-64
> and not on x86. My guess is that the problem lies in gcc. Even building with
> CXX="g++ --coverage" and CC="gcc --coverage" results in the same crashes.

They are occasionally omitted on all platforms. Thanks for the info,

[Tglek (tglek)]

files the x86_64 gcc bug as http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43825

[Tglek (tglek)]

Created an attachment (id=440627)
pgo-friendly compiler flags

this plus fix in bug 560897 = working pgo on x86_64(and likely gcc 4.5)

[Mh+mozilla (mh+mozilla)]

-fprofile-correction in optim flags doesn't sound right

[Tglek (tglek)]

(In reply to comment #76)
> -fprofile-correction in optim flags doesn't sound right

it's not something that would land, just enough to show that pgo works. Sticking those flags in right places is the right thing(tm).

[Tglek (tglek)]

Created an attachment (id=442123)
configury

[Virus-found (virus-found)]

(In reply to comment #78)
> Created an attachment (id=442123) [details]
> configury

The patch is absolutely amazing. Works even with my complicated .mozconfig. Thank you very much! We all have been waiting for it.

[Kwinz (ldm)]

New patch upsteam.
https://bugzilla.mozilla.org/show_bug.cgi?id=418866
Let's get PGO working now!

[Mh+mozilla (mh+mozilla)]

(From update of attachment 442123)
Pushed as http://hg.mozilla.org/mozilla-central/rev/7a6a20da79ae

Should this bug be closed or is actually enabling pgo build on build servers part of this bug ?

[Tglek (tglek)]

(In reply to comment #80)
> (From update of attachment 442123 [details])
> Pushed as http://hg.mozilla.org/mozilla-central/rev/7a6a20da79ae
>
> Should this bug be closed or is actually enabling pgo build on build servers
> part of this bug ?

Thanks. We'll turn it on via bug 559964.

[Evgeny Burmentyev (vir-found)]

Hm, it fails to compile now. Regression range is http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=ae1773250d39&tochange=5b3604a3cfbe

gcc -o now.o -c -fomit-frame-pointer -Wall -pthread -march=core2 -O2 -pipe -fPIC -UDEBUG -DMOZILLA_CLIENT=1 -DNDEBUG=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE_VISIBILITY_PRAGMA=1 -DXP_UNIX=1 -D_GNU_SOURCE=1 -DHAVE_FCNTL_FILE_LOCKING=1 -DLINUX=1 -Di386=1 -DHAVE_LCHOWN=1 -DHAVE_STRERROR=1 -D_REENTRANT=1 -DFORCE_PR_LOG -D_PR_PTHREADS -UHAVE_CVAR_BUILT_ON_SEM -fprofile-use -fprofile-correction -Wcoverage-mismatch -freoder-blocks-and-partition /home/virus_found/abs/firefox/src/mozilla-central-build/nsprpub/config/now.c
cc1: error: unrecognized command line option "-freoder-blocks-and-partition"
make[5]: *** [now.o] Error 1

[Tglek (tglek)]

(In reply to comment #82)

> cc1: error: unrecognized command line option "-freoder-blocks-and-partition"
> make[5]: *** [now.o] Error 1

Sounds like you configured with one compiler and are building with an older one.

[Evgeny Burmentyev (vir-found)]

(In reply to comment #83)
> (In reply to comment #82)
>
> > cc1: error: unrecognized command line option "-freoder-blocks-and-partition"
> > make[5]: *** [now.o] Error 1
>
> Sounds like you configured with one compiler and are building with an older
> one.

I'm sorry, I don't get it. What do you mean by "configured"? I configure with autoconf-2.13.

[Mh+mozilla (mh+mozilla)]

(In reply to comment #83)
> (In reply to comment #82)
>
> > cc1: error: unrecognized command line option "-freoder-blocks-and-partition"
> > make[5]: *** [now.o] Error 1
>
> Sounds like you configured with one compiler and are building with an older
> one.

It just looks like there is a typo... reoder vs. reorder.

[Evgeny Burmentyev (vir-found)]

(In reply to comment #85)
> It just looks like there is a typo... reoder vs. reorder.

Yes, this turns out to be the reason. This was introduced in http://hg.mozilla.org/mozilla-central/rev/b5b016bb7c91

[Evgeny Burmentyev (vir-found)]

Created attachment 449604
fixes two typos, introduced in b5b016bb7c91 commit

[Evgeny Burmentyev (vir-found)]

Comment on attachment 449604
fixes two typos, introduced in b5b016bb7c91 commit

I'm not permitted to change anything in this bug, but adding comments and patches, so this needs to be checked in.

[Ted Mielczarek (ted-mielczarek)]

Comment on attachment 449604
fixes two typos, introduced in b5b016bb7c91 commit

This is fallout from bug 564851. My patch there is correct, but somehow I screwed it up while landing, apparently. I checked in a fix in NSPR CVS.

[Chris Coulson (chrisccoulson)]

This is something we are going to try and get working in Natty

[Ted Mielczarek (ted-mielczarek)]

We have finally flipped this on for our official Firefox Linux builds using GCC 4.5, see https://bugzilla.mozilla.org/show_bug.cgi?id=559964 . Doing PGO for XULRunner would be a little different, since you'd need a different profiling script.

[sam tygier (samtygier)]

official firefox 6 builds now have PGO. ubuntu 6.0+build1+nobinonly-0ubuntu0.11.04.1 does not have it.

[sam tygier (samtygier)]

I ran the peacekeeper benchmark on the official mozilla ff6 build, and ubuntu 6.0+build1+nobinonly-0ubuntu0.11.04.1. on an intel atom N270, with 2GB ram and an SSD.

official ff6 1002
rendering 761
soc networking 844
complex graphics 3687
data 1401
DOM 722
text 1557

ubuntu ff6 888
rendering 544
soc networking 678
complex graphics 3139
data 1436
DOM 618
text 1691

so a bit more than a 10% improvement for me on low end hardware.

[madbiologist (me-again)]

Is PGO enabled in firefox 7.0~b4+build2+nobinonly-0ubuntu1 in the Ubuntu 11.10 repositories?

[madbiologist (me-again)]

Is PGO enabled in firefox 7.0+build2+nobinonly-0ubuntu4 in the Ubuntu 11.10 repositories?

[sam tygier (samtygier)]

no, still built with -Os. you can check by putting about:buildconfig in the url bar.

[greg (grigorig)]

Firefox' Linux release builds have been using PGO for some time now. The difference is quite noticeable, especially on slow systems, such as netbooks. Why is this not used by Ubuntu yet? Firefox is Ubuntu's standard browser and users should have the best experience possible.

What is missing to get this working? Is it still possible to get PGO builds into Precise? I'm willing to help.

[madbiologist (me-again)]

Note that we should also build with -O3 to get the best out of PGO - see https://bugzilla.mozilla.org/show_bug.cgi?id=590181

@Sam Tygier - re comment #135 - unless I am understanding what I've read in the Mozilla bugs incorrectly, -Os seems to be an alternative to -O1/O2/O3, and as such, is separate to PGO. Is this correct, and if so what should I look for in about:buildconfig to confirm the use of PGO? I couldn't find this in the Mozilla bugs.

[greg (grigorig)]

You need to check for -fprofile-use.

[Chris Coulson (chrisccoulson)]

Fixed in quantal

[greg (grigorig)]

Are we going to see PGO in updated Firefox releases for precise?

[Chris Coulson (chrisccoulson)]

You're getting a bit ahead of yourself, it hasn't even built in quantal yet. Perhaps ask me again in 4 months when it's had some proper testing there first

[Chris Coulson (chrisccoulson)]

I'm turning this back off again due to continual build failures that are only reproducible on the buildd's, and because the the PPA builders are pretty much incapable of building it to help investigate

[Chris Coulson (chrisccoulson)]

Actually, scratch that. We get the same build failure in thunderbird without PGO.

[Chris Coulson (chrisccoulson)]

I'm backing this out of quantal. The benchmarks show negligible benefit of turning on PGO in most tests, and there isn't really any perceivable difference at all - certainly not enough to justify a 100% increase in build time

[madbiologist (me-again)]

Chris - thanks for your efforts on this. I am a little surprised to hear that the benchmarks show negligible benefit of turning on PGO in most tests, given what I have read previously in this bug and in the Mozilla bug reports. However I am not an expert so I guess it is possible. I just have one question - did you also build with -O3 as per comment #137?

[Chris Coulson (chrisccoulson)]

Yes, the build system automatically sets that when you build with PGO

[zamazan4ik (zamazan4ik)]

Regarding PGO benefits for Firefox, there is a thread: https://groups.google.com/g/mozilla.dev.platform/c/wwO48xXFx0A/m/ztg4i0DYAAAJ that shows measurable improvements from PGO.

Anyway, since the issue is quite old. Is PGO still disabled for Firefox in Ubuntu repositories? Maybe things have changed during these years...

[Sebastien Bacher (seb128)]

Yes, things changed since the report, Ubuntu provides firefox as a snap now and PGO has been enabled on the amd64 build for a while now
https://github.com/canonical/firefox-snap/commit/7fc10a451