Firefox crashes at start on armv7L after 55.0.1 update

[Jim Gregory]

Firefox always crashes when launched after the 55.0.1 update on an Orange Pi PC Plus (a single-board computer similar to a Raspberry Pi), even in safe mode.

I did a fresh install of Armbian (a Ubuntu Xenial 16.04 re-spin for ARM single-board computer) on a similar board (Orange Pi Plus 2e), installed Firefox and experienced the same problem--it won't load without crashing.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: firefox 55.0.1+build2-0ubuntu0.16.04.2
Uname: Linux 3.4.113-sun8i armv7l
AddonCompatCheckDisabled: False
AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.25.
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: armhf
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC1: jim 1138 F.... pulseaudio
 /dev/snd/controlC0: jim 1138 F.... pulseaudio
BuildID: 20170814194718
Card0.Amixer.info:
 Card hw:0 'audiocodec'/'audiocodec'
   Mixer name : ''
   Components : ''
   Controls : 12
   Simple ctrls : 12
Card1.Amixer.info:
 Card hw:1 'sndhdmi'/'sndhdmi'
   Mixer name : ''
   Components : ''
   Controls : 1
   Simple ctrls : 1
Card1.Amixer.values:
 Simple mixer control 'hdmi audio format Function',0
   Capabilities: enum
   Items: 'null' 'pcm' 'AC3' 'MPEG1' 'MP3' 'MPEG2' 'AAC' 'DTS' 'ATRAC' 'ONE_BIT_AUDIO' 'DOLBY_DIGITAL_PLUS' 'DTS_HD' 'MAT' 'WMAPRO'
   Item0: 'pcm'
Channel: Unavailable
CurrentDesktop: XFCE
Date: Thu Aug 17 05:37:00 2017
Extensions: extensions.sqlite corrupt or missing
ForcedLayersAccel: False
IncompatibleExtensions: Unavailable (corrupt or non-existant compatibility.ini or extensions.sqlite)
IpRoute:
 default via 192.168.10.1 dev eth0
 default via 192.168.10.1 dev wlan0 proto static metric 600
 169.254.0.0/16 dev eth0 scope link metric 1000
 192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.107
 192.168.10.0/24 dev wlan0 proto kernel scope link src 192.168.10.108 metric 600
Locales: extensions.sqlite corrupt or missing
PciMultimedia:

PciNetwork:

Profiles: Profile0 (Default) - LastVersion=55.0.1/20170814194718
RfKill:
 0: phy0: Wireless LAN
  Soft blocked: no
  Hard blocked: no
RunningIncompatibleAddons: False
SourcePackage: firefox
Themes: extensions.sqlite corrupt or missing
UpgradeStatus: No upgrade log present (probably fresh install)

[LarryWebber (larrywebber)]

I have both an Odroid XU4 and crouton xenial (running on Asus C100) which are arm based and Firefox also does as described - Intel based machines that I have do not exhibit this issue

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Wouter Lippe (wouterlippe)]

I got the same problem on Raspberry Pi3 with Ubuntu Mate.

[herrtimson (herrtimson)]

I run into this on 14.04 with armhf / arm device, it is an old ac100 netbook. Is there any kind of upstream bug at mozillas bugtracker?

[Garry Trethewey (garrytreth)]

I also got the same problem on Raspberry Pi3 with Ubuntu Mate 16.04 as soon as I did the first update.
https://ubuntu-mate.community/t/firefox-55-0-2-doesnt-start-crashes-on-ubuntu-mate-raspberrypi-3/14637 has some workarounds, if anybody thinks to look there.

[Adam Smith (adamsmith)]

This is probably an unhelpful post, but ff 55 works on arm64. So if you have a pi 3 and are desperate for the latest ff then *ubuntu arm64 is a possibility. Same machine crashes with lubuntu 16.04 armhf.

[Steve Desmond (vtsv)]

Same problem, Ubuntu MATE 16.04 on ASUS Chromebook Flip (C100P)

[in8sworld (n8berry)]

Same problem on ASUS Chromebook C201 running Xenial under Crouton (armhf).
firefox 55.0.1 and 55.0.2 (both published 8/17) crash immediately at launch.

firefox 52.0.2 from 8/15 still works for me:
https://launchpad.net/ubuntu/xenial/armhf/firefox/52.0.2+build1-0ubuntu0.16.04.1
firefox_52.0.2+build1-0ubuntu0.16.04.1_armhf.deb (40.4 MiB)

[Adam Smith (adamsmith)]

On the Mozilla bug people are saying they had 54 working, but there was no 53 or 54 on armhf in Ubuntu? That is why people are rolling back to 52 isn't it?

[herrtimson (herrtimson)]

No, people go back to 52 because it is working and it is under active developement, hence gets sec. updates. firefox 53 and 54 branches are stalled, they are very likely vulnerable.

[Adam Smith (adamsmith)]

Not in Ubuntu. The link above is from March.

[herrtimson (herrtimson)]

On a rpi2 I tested Arch Linux recently, and their firefox-55.0.2 binary does work like a charm. Which makes me wonder if this is a bug somewhere in the toolchain (they use bleeding edge very latest gcc/glibc/etc.), or if the culprit could be found with the configure options.

How can we find out the build/configure options the ubuntu team uses?

On the other hand, I tested a build with gentoo von armv7 and ran into a segfault with 55.0.2 as well.

[herrtimson (herrtimson)]

Today I build firefox 56.0 beta 12 which segfaults as well (on gentoo-armv7a-unknown-hardfp)

The toolchain I built with is gcc-5.4.0, binutils-2.28.1, glibc-2.23

So I doubt this will solve itself :( and would like to opt for a downgrade of firefox to esr for as long as it takes to solve this.

[Launchpad Janitor (janitor)]

This bug was fixed in the package firefox - 55.0.2+build1-0ubuntu4

---------------
firefox (55.0.2+build1-0ubuntu4) artful; urgency=medium

  * New usptream stable release (55.0.2build1)

  [ Rico Tzschichholz ]
  * Refresh patches
    - update debian/patches/revert-upstream-search-engine-changes.patch
  * Add multiple ppc64/arm/s390x build fixes
    - debian/patches/build-ppc64.patch
    - debian/patches/build-ppc64-s390x-curl.patch
    - debian/patches/build-ppc64-s390x-nss.patch
    - debian/patches/build-ppc64-s390x-rust.patch
  * Fix Spidermonkey build with no jit backend
    - debian/patches/fix-no-jit-backend.patch
  * Do not use compile-time page size for PowerPC
    - debian/patches/ppc-no-static-sizes.patch
  * Update debian/build/create-tarball.py to create xz-tarballs
  * Refresh patches
    - update debian/patches/ubuntu-search-defaults.patch
    - update debian/patches/ubuntu-ua-string-changes.patch
    - update debian/patches/unity-menubar.patch
    - update debian/patches/revert-upstream-search-engine-changes.patch
    - update debian/patches/normalize-distribution-searchplugins.patch
  * Remove obsolete --enable-gio option
    - update debian/config/mozconfig.in
  * Refresh shipped locales
    - update debian/config/locales.shipped
    - update debian/control
  * Don't let debhelper clean Cargo.toml.orig files which are actually required

  [ Chris Coulson ]
  * Refresh patches
    - update debian/patches/unity-menubar.patch
  * Disable ALSA backend for Firefox 55 because it makes the build failure.
    This will remain disabled until somebody steps up to maintain it
  * Fix a build failure in cubeb-pulse-rs on architectures where char is
    unsigned
    - add debian/patches/cubeb-pulse-rs-fix.patch
    - update debian/patches/series
  * Refactor nsNativeMenuService ownership and shutdown behaviour - the only
    reference to it now is held by the service manager, and the global instance
    pointer is no longer cleared during the xpcom-shutdown notification. This
    fixes a shutdown UAF that appeared in the current Firefox version, where
    nsWindow and the associated nsMenuBar can be destroyed after xpcom-shutdown,
    when it was too late to clean up properly.
    - update debian/patches/unity-menubar.patch
  * Hopefully fix LP: #1711337 by building the armhf build with
    -fno-schedule-insns to work around a compiler bug
  * Backport a couple of upstream breakpad-client fixes to unbreak the build
    with glibc 2.26
    - add debian/patches/use-ucontext_t-in-breakpad-client.patch
    - add debian/patches/use-ucontext_t-in-breakpad-client_2.patch
    - update debian/patches/series

 -- Chris Coulson <email address hidden> Mon, 07 Aug 2017 11:43:19 +0100

[lavezarez (lavezarez)]

Firefox (55.0.2+build1-0ubuntu0.16.04.1) still crashing / unable to open on my Asus Chromebook running crouton target lxde

[herrtimson (herrtimson)]

That is because, as far as I understand it, this bug hasn't been resolved for any ubuntu version lower than artful (17.10), for reasons unexplained. It might be related to the release of 56.0 a day ago, so it is best to wait a few days and hopefully the fix will be backported to esr branches.

@mozilla team: do you happen to have an upstream gcc bug url, since this is a compiler bug?

[Karim Kronfli (kkronfli)]

Any idea when the bug will be fixed in 56?

[Poil (poil)]

Please, fix it on Ubuntu 16.04 (latest Release and latest Beta are KO) !

[herrtimson (herrtimson)]

I just tested, bug is not fixed in 16.04 and not in 14.04.

[Adam Smith (adamsmith)]

This is not fixed in 17.04 either. Tried 55 which supposedly has the fix above, and 56.

[Adam Smith (adamsmith)]

Sorry that should of been 17.10 (artful ardvark)!

[Hans Joachim Desserud (hjd)]

Tagged also affected releases based on comment #20 and comment #22.

[Jang Ryeol (bekker)]

57 on 17.10 does not work either.

[Damian Christey (damian-christey)]

Please change status back to "confirmed". The released fix did not work, as evidenced by the ongoing duplicate bug reports.

[Saeid Ghazagh (sghazagh)]

The latest package installs on 16.04.3 does not work either.
I only could get it working by installing the "firefox_57.0.1+build2-0ubuntu0.16.04.1_armhf.deb" package.
Anyone has got if there is any latest fixed version?

[Chituc Georgian (dianaxxyyzz)]

So "firefox_57.0.1+build2-0ubuntu0.16.04.1_armhf.deb" is working ?

[Karim Kronfli (kkronfli)]

Confirmed as still broken on Raspberry Pi3

Version: 57.0.1+build2-0ubuntu0.16.04.1

[Karim Kronfli (kkronfli)]

The fix was never ported into v56 or newer

[James Donald (jdonald)]

That fix was:
> * Hopefully fix LP: #1711337 by building the armhf build with
> -fno-schedule-insns to work around a compiler bug
?

According to https://launchpadlibrarian.net/351233449/buildlog_ubuntu-xenial-armhf.firefox_57.0.3+build1-0ubuntu0.16.04.1_BUILDING.txt.gz it's building with:
> --enable-optimize=-g -O2 -fno-schedule-insns

So it looks like the fix was indeed ported, but given the word "hopefully", but was it ever confirmed?

Just like Adam Smith with version 55, arm64 appears to be fine. And just like herrtimson with version 55, even armv7 works on Arch Linux but still crashes on Debian/Ubuntu: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=150438&start=50

[Lars Nyqvist (ulos)]

firefox 57.0.3+build1-0ubuntu0.17.10.1 still fails.

On 12/30/17, James Donald <email address hidden> wrote:
> That fix was:
>> * Hopefully fix LP: #1711337 by building the armhf build with
>> -fno-schedule-insns to work around a compiler bug
> ?
>
> According to
> https://launchpadlibrarian.net/351233449/buildlog_ubuntu-xenial-armhf.firefox_57.0.3+build1-0ubuntu0.16.04.1_BUILDING.txt.gz
> it's building with:
>> --enable-optimize=-g -O2 -fno-schedule-insns
>
> So it looks like the fix was indeed ported, but given the word
> "hopefully", but was it ever confirmed?
>
> Just like Adam Smith with version 55, arm64 appears to be fine. And just
> like herrtimson with version 55, even armv7 works on Arch Linux but
> still crashes on Debian/Ubuntu:
> https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=150438&start=50
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1733340).
> https://bugs.launchpad.net/bugs/1711337
>
> Title:
> Firefox crashes at start on armv7L after 55.0.1 update
>
> Status in Mozilla Firefox:
> Confirmed
> Status in firefox package in Ubuntu:
> Fix Released
>
> Bug description:
> Firefox always crashes when launched after the 55.0.1 update on an
> Orange Pi PC Plus (a single-board computer similar to a Raspberry Pi),
> even in safe mode.
>
> I did a fresh install of Armbian (a Ubuntu Xenial 16.04 re-spin for
> ARM single-board computer) on a similar board (Orange Pi Plus 2e),
> installed Firefox and experienced the same problem--it won't load
> without crashing.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 16.04
> Package: firefox 55.0.1+build2-0ubuntu0.16.04.2
> Uname: Linux 3.4.113-sun8i armv7l
> AddonCompatCheckDisabled: False
> AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.25.
> ApportVersion: 2.20.1-0ubuntu2.10
> Architecture: armhf
> AudioDevicesInUse:
> USER PID ACCESS COMMAND
> /dev/snd/controlC1: jim 1138 F.... pulseaudio
> /dev/snd/controlC0: jim 1138 F.... pulseaudio
> BuildID: 20170814194718
> Card0.Amixer.info:
> Card hw:0 'audiocodec'/'audiocodec'
> Mixer name : ''
> Components : ''
> Controls : 12
> Simple ctrls : 12
> Card1.Amixer.info:
> Card hw:1 'sndhdmi'/'sndhdmi'
> Mixer name : ''
> Components : ''
> Controls : 1
> Simple ctrls : 1
> Card1.Amixer.values:
> Simple mixer control 'hdmi audio format Function',0
> Capabilities: enum
> Items: 'null' 'pcm' 'AC3' 'MPEG1' 'MP3' 'MPEG2' 'AAC' 'DTS' 'ATRAC'
> 'ONE_BIT_AUDIO' 'DOLBY_DIGITAL_PLUS' 'DTS_HD' 'MAT' 'WMAPRO'
> Item0: 'pcm'
> Channel: Unavailable
> CurrentDesktop: XFCE
> Date: Thu Aug 17 05:37:00 2017
> Extensions: extensions.sqlite corrupt or missing
> ForcedLayersAccel: False
> IncompatibleExtensions: Unavailable (corrupt or non-existant
> compatibility.ini or extensions.sqlite)
> IpRoute:
> default via 192.168.10.1 dev eth0
> default via 192.168.10.1 dev wlan0 proto static metric 600
> 169.254.0.0/16 dev eth0 scope link metric 1000
> 192.168.10.0/24 dev eth0 proto kernel scope link src 1...

[herrtimson (herrtimson)]

Not fixed for trusty, it still fails with the same error over and over again.

I do understand that this free software, and that the ubuntu devs are unpayed volunteers basically. However, this is one is a big bummer for so many people out there, so can you please throw some ressources at it to solve this eventually?

It is possible to solve it, see the arch linux binary.

[Damian Christey (damian-christey)]

I think this bug can be fixed, but the first step is getting someone with the required permissions to acknowledge that it exists, and change the status back to confirmed. With its current status, it probably isn't showing up on the radar of any Ubuntu developers.

[herrtimson (herrtimson)]

Maybe someone could ping the team in the irc?

By the way, https://bugs.launchpad.net/ubuntu/+source/firefox/+bug/1732954 is a duplicate, but it has some additional information in terms of a backtrace!

[Lars Nyqvist (ulos)]

firefox-57.0.4 still fails

cloudsto@RIKOMAGIC:~$ firefox -g
GNU gdb (Ubuntu 8.0.1-0ubuntu1) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/firefox/firefox...Reading symbols from
/usr/lib/debug//usr/lib/firefox/firefox...done.
done.
(gdb) run
Starting program: /usr/lib/firefox/firefox
Cannot parse expression `.L1200 4@r4'.
warning: Probes-based dynamic linker interface failed.
Reverting to original interface.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0xb17ff440 (LWP 15197)]
[New Thread 0xadf28440 (LWP 15198)]
[Thread 0xadf28440 (LWP 15198) exited]
[New Thread 0xadf28440 (LWP 15199)]
[New Thread 0xaccb3440 (LWP 15200)]
[New Thread 0xac4b3440 (LWP 15201)]
[New Thread 0xabaff440 (LWP 15202)]
[New Thread 0xab2ff440 (LWP 15203)]
[New Thread 0xab0ff440 (LWP 15204)]
[New Thread 0xaaeff440 (LWP 15205)]
[New Thread 0xaacff440 (LWP 15206)]
[New Thread 0xaaaff440 (LWP 15207)]
[New Thread 0xaa8ff440 (LWP 15208)]
[New Thread 0xaa6ff440 (LWP 15209)]
[New Thread 0xaa4ff440 (LWP 15210)]
[New Thread 0xa9dff440 (LWP 15215)]
[New Thread 0xa95ff440 (LWP 15216)]
[New Thread 0xa8dff440 (LWP 15217)]
[New Thread 0xa85ff440 (LWP 15218)]
[New Thread 0xa7cff440 (LWP 15219)]
[New Thread 0xb19d9440 (LWP 15221)]
[Thread 0xa7cff440 (LWP 15219) exited]
[New Thread 0xa7cff440 (LWP 15223)]
[New Thread 0xa6191440 (LWP 15224)]
[New Thread 0xa5724440 (LWP 15225)]
[New Thread 0xa4f24440 (LWP 15226)]
[New Thread 0xa4594440 (LWP 15229)]
[New Thread 0xa3d94440 (LWP 15230)]
[New Thread 0xa3594440 (LWP 15231)]
[New Thread 0xa2d94440 (LWP 15232)]
[New Thread 0xa23a9440 (LWP 15233)]
[New Thread 0xa1ba9440 (LWP 15234)]
[New Thread 0xa13a9440 (LWP 15235)]
[New Thread 0xa0ba9440 (LWP 15236)]
[Thread 0xa0ba9440 (LWP 15236) exited]
[Thread 0xa13a9440 (LWP 15235) exited]
[Thread 0xa1ba9440 (LWP 15234) exited]
[New Thread 0xa0ba9440 (LWP 15237)]
[New Thread 0xa13a9440 (LWP 15238)]
[New Thread 0xa1ba9440 (LWP 15239)]
[New Thread 0x9fa95440 (LWP 15240)]
[New Thread 0x9f295440 (LWP 15241)]
[New Thread 0x9ea95440 (LWP 15242)]
[New Thread 0x9e295440 (LWP 15243)]
[New Thread 0x9da95440 (LWP 15244)]
[New Thread 0x9d295440 (LWP 15245)]
[New Thread 0x9ca95440 (LWP 15246)]
[New Thread 0x9c0ff440 (LWP 15247)]
[New Thread 0x9b8ff440 (LWP 15248)]
[New Thread 0x9afff440 (LWP 15249)]
[New Thread 0x9a7ff440 (LWP 15250)]
[Thread 0x9c0ff440 (LWP 15247) exited]
[Thread 0x9b8ff440 (LWP 15248) exited]
[New Thread 0x999ff440 (LWP 15251)]
[New Thread 0x9...

[Karim Kronfli (kkronfli)]

This bug is not fixed properly

[James Donald (jdonald)]

Thanks for providing the stack trace, Lars.

This shows the illegal instruction is happening in Skia, specifically SkJumper's implementation of _sk_xor__vfp4. To run with Skia disabled, edit your user profile at ~/.mozilla/firefox/*.default/prefs.js and add the following line:

user_pref("gfx.content.azure.backends", "");

Firefox 32-bit will then launch on Raspbian Stretch. Or if you have more than one user, you can instead edit /usr/lib/firefox/defaults/pref/vendor-gre.js

I expect rendering performance is taking a hit with Skia disabled. For a proper fix we should figure out what's going on with the illegal instruction(s) in SkJumper_generated.S. This file is auto-generated so from what I can tell the armv7 version isn't readily available via git. I have yet to set up a build environment for Firefox, but if someone here has done so for Arch Linux and another for Ubuntu, we can diff and track this down.

[levente (leventelist)]

I have the same issue, however, I don't have the same stack trace.

$ firefox -g

(gdb) run
Starting program: /usr/lib/firefox/firefox

Program received signal SIGSEGV, Segmentation fault.
0xb6fd9dde in ?? () from /lib/ld-linux-armhf.so.3
(gdb) where
#0 0xb6fd9dde in ?? () from /lib/ld-linux-armhf.so.3
#1 0xb6fd9df6 in ?? () from /lib/ld-linux-armhf.so.3
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

# dpkg -l | grep firefox
ii firefox 57.0.4+build1-0ubuntu0.16.04.1 armhf Safe and easy web browser from Mozilla
ii firefox-dbg 57.0.4+build1-0ubuntu0.16.04.1 armhf Safe and easy web browser from Mozilla - debug symbols

Anyone knows how can I get a corrupt stack?

[James Donald (jdonald)]

levente,

I'm not sure how Lars and Sam Tygier (#1732954) are getting useful backtraces. Might have something to do with running 17.10?

> ii firefox 57.0.4+build1-0ubuntu0.16.04.1 armhf

I see you're using 16.04. I should have mentioned that the workaround of disabling Skia is something I verified using the 14.04 Trusty package (on Raspbian). The flag also seems sufficient for Firefox to run on 17.10, but from what I can tell there's a second crash specific to the 16.04 build. To avoid that unknown, you can download and install the Trusty .deb (via dpkg -i) even if your system is 16.04 Xenial: https://launchpad.net/ubuntu/trusty/armhf/firefox/57.0.4+build1-0ubuntu0.14.04.1

[Mathias (go4)]

This works with Ubuntu Mate 17.10 (armhf) on a Raspberry Pi 3

sudo apt-get purge firefox
wget http://launchpadlibrarian.net/352602073/firefox_57.0.4+build1-0ubuntu0.14.04.1_armhf.deb
sudo dpkg -i firefox_57.0.4+build1-0ubuntu0.14.04.1_armhf.deb
echo 'user_pref("gfx.content.azure.backends", "");' >> ~/.mozilla/firefox/*.default/prefs.js

Many thanks, James Donald.

[in8sworld (n8berry)]

Confirmed that firefox_57.0.4+build1-0ubuntu0.14.04.1_armhf.deb is working and without the prefs.js edit mentioned above on my Xenial (16.04.3 LTS) under Crouton on an ASUS C201 chromebook.

[herrtimson (herrtimson)]

Sadly, this

>
sudo apt-get purge firefox
wget http://launchpadlibrarian.net/352602073/firefox_57.0.4+build1-0ubuntu0.14.04.1_armhf.deb
sudo dpkg -i firefox_57.0.4+build1-0ubuntu0.14.04.1_armhf.deb
echo 'user_pref("gfx.content.azure.backends", "");' >> ~/.mozilla/firefox/*.default/prefs.js
<

doesn't work for the case of 14.04. It seems to bypass the initially segfault, but to the cost of another one.

[Lars Nyqvist (ulos)]

Firefox 58 fails as before and user_pref trick does not work any more!!
I installed 58 , rebooted and tested

~$ fgrep gfx. /home/cloudsto/.mozilla/firefox/ymtjpt2g.default-1510054306584-1514625894232/prefs.jsuser_pref("gfx.content.azure.backends",
"");
user_pref("services.blocklist.gfx.checked", 1516706246);

~$ firefox -g
GNU gdb (Ubuntu 8.0.1-0ubuntu1) 8.0.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/firefox/firefox...Reading symbols from
/usr/lib/debug/.build-id/dc/cf886d995205c45d42314d4e755178a87359d9.debug...done.
done.
(gdb) run
Starting program: /usr/lib/firefox/firefox
Cannot parse expression `.L1200 4@r4'.
warning: Probes-based dynamic linker interface failed.
Reverting to original interface.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0xb18b0440 (LWP 1554)]
[New Thread 0xadeb5440 (LWP 1555)]
[Thread 0xadeb5440 (LWP 1555) exited]
[New Thread 0xadeb5440 (LWP 1556)]
[New Thread 0xacaff440 (LWP 1557)]
[New Thread 0xac2ff440 (LWP 1558)]
[New Thread 0xab9ff440 (LWP 1559)]
[New Thread 0xab1ff440 (LWP 1560)]
[New Thread 0xaafff440 (LWP 1561)]
[New Thread 0xaadff440 (LWP 1562)]
[New Thread 0xaabff440 (LWP 1563)]
[New Thread 0xaa9ff440 (LWP 1564)]
[New Thread 0xaa7ff440 (LWP 1565)]
[New Thread 0xaa5ff440 (LWP 1566)]
[New Thread 0xaa3ff440 (LWP 1567)]
[New Thread 0xa9dff440 (LWP 1568)]
[New Thread 0xa95ff440 (LWP 1569)]
[New Thread 0xa8dff440 (LWP 1570)]
[New Thread 0xa85ff440 (LWP 1571)]
[New Thread 0xa7aff440 (LWP 1572)]
[New Thread 0xa71ff440 (LWP 1574)]
[New Thread 0xa7d97440 (LWP 1575)]
[New Thread 0xa69ff440 (LWP 1576)]
[Thread 0xa7aff440 (LWP 1572) exited]
[New Thread 0xa7aff440 (LWP 1577)]
[New Thread 0xa4cb6440 (LWP 1578)]
[New Thread 0xa44b6440 (LWP 1579)]
[New Thread 0xa3cb6440 (LWP 1580)]
[New Thread 0xa321f440 (LWP 1581)]
[New Thread 0xa2a1f440 (LWP 1582)]
[New Thread 0xa221f440 (LWP 1583)]
[New Thread 0xa1a1f440 (LWP 1584)]
[New Thread 0xa0eda440 (LWP 1585)]
[New Thread 0xa06b1440 (LWP 1586)]
[New Thread 0x9feb1440 (LWP 1587)]
[New Thread 0x9f6b1440 (LWP 1588)]
[Thread 0x9f6b1440 (LWP 1588) exited]
[Thread 0x9feb1440 (LWP 1587) exited]
[Thread 0xa06b1440 (LWP 1586) exited]
[New Thread 0xa06b1440 (LWP 1589)]
[New Thread 0x9f6b1440 (LWP 1590)]
[New Thread 0x9feb1440 (LWP 1591)]
[New Thread 0x9e695440 (LWP 1592)]
[New Thread 0x9de95440 (LWP 1593)]
[New Thread 0x9d695440 (LWP 1594)]
[New Thread 0x9ce95440 (LWP 1595)]
[New Thread 0x9c695440 (LWP 1596)]
[New Thread 0x9be95440...

[James Donald (jdonald)]

Darn, and I see the same error with Firefox 59. Your stack trace still shows the illegal instruction in sk_xor__vfp4(). It seems like something is now running Skia even when it's disabled in the config.

Among rendering differences, from the Firefox 58 blog post there's those changes for OMTP which could have affected this: https://mozillagfx.wordpress.com/2017/12/05/off-main-thread-painting/

If it's now going into Skia when it shouldn't that's a Firefox 58 regression. It would be somewhat of a bug affecting non-armhf platforms too so we could finally get the attention of the Mozilla team.

Meanwhile the bad assembly in SkJumper_generated.S still seems like an Ubuntu-specific problem. Is anyone able to reach the Ubuntu dev team on IRC so that can set this bug's status back to "confirmed"?

[Chituc Georgian (dianaxxyyzz)]

I read on chromium forum that they had exacty this problem too , with the crash on skia and this problem do not appear if they crosscompiled chromium , so maybe the solution is to cross compile firefox too .

[James Donald (jdonald)]

Various notes:

Good point by Chituc. I had assumed this was already cross-compiled, but according to the Ubuntu build log:
> checking whether cross compiling... no

Who here knows how the Arch Linux armhf package is built?

It's also still possible that a newer toolchain could fix things. I read somewhere that Ubuntu 18.04 Bionic would be upgrading to gcc 7. Anyone tried Firefox 58 or 59 on there?

The SkJumper_generated source is available on Mercurial: https://hg.mozilla.org/mozilla-central/raw-file/tip/gfx/skia/skia/src/jumper/SkJumper_generated.S

It's possible to take a closer look at the "illegal instruction" with firefox -g. At the crash point in gdb, run: layout asm; set arm fallback-mode arm; tui disable; disas /r <starting instruction>,<ending instruction>. Initially it gets confused about the instruction layout, but after trying different ranges it'll eventually decode properly:

(gdb) disas /r 0xf458bb28,0xf458bb3e
Dump of assembler code from 0xf458bb28 to 0xf458bb3e:
   0xf458bb28: 92 2c 44 f2 vfma.f32 d18, d20, d2
   0xf458bb2c: 93 3c 44 f2 vfma.f32 d19, d20, d3
   0xf458bb30: b0 01 20 f2 vorr d0, d16, d16
   0xf458bb34: b1 11 21 f2 vorr d1, d17, d17
   0xf458bb38: b2 21 22 f2 vorr d2, d18, d18
   0xf458bb3c: b3 31 23 f2 vorr d3, d19, d19
End of assembler dump.

However, those are exactly the instructions specified for _sk_xor__vfp4 in SkJumper_generated.S. It's hard to see how the compiler or binutils could be directly responsible for generating an illegal instruction here.

You can actually hexedit libxul.so and change those four vorr instructions into NOPs. Firefox then makes it past this point but crashes obscurely elsewhere, with different crash signatures on Firefox 58 vs 59.

Firefox arm64 (Pi 3 or other 64-bit board required) 58 and 59 still work.

[Chituc Georgian (dianaxxyyzz)]

They at chromium say they use clang when they cross compile chromium .And they had exactly same problem like firefox have ,when they build chromium native using gcc .
You can read a irc log :

--------------------
[16:17:16] <msanchez> This has been puzzling us for a few weeks already because we can't get that crash when we cross compile chromium, but only when running a version built natively in an arm7 machine
[16:18:55] <lizeb> Does gdb tell you which instruction is causing trouble? It should
[16:19:36] <msanchez> the only differences we could spot are that in the native build we pass use_sysroot=false and is_clang=false (we build with gcc 4.9), while in the cross-build we pass is_clang=true and use_sysroot=true
[16:19:54] <msanchez> #0 0xb629b9e6 in _sk_xor__vfp4 () from /usr/lib/chromium-browser/libskia.so
[16:20:00] <msanchez> is that what you mean?
[16:20:17] <msanchez> sorry, I'm not deep in ARM-internals, feel free to ask anything even if it sounds dumb :)
[16:21:05] <lizeb> try "disas 0xb629b9e6,0xb629b100"
[16:21:27] <lizeb> it will give you the assembly at these addresses
[16:21:49] <lizeb> since the issue is "illegal instruction", it might be that the instruction executed isn't supported by your target
[16:22:08] <msanchez> that makes sense
[16:22:12] * msanchez boots up the ARM device
[16:22:34] <msanchez> At the moment my theories were pointing to some compiler-specific thing
[16:22:49] <msanchez> because of the difference between using clang (cross-build) vs gcc 4.9 (native build)
[16:23:30] <lizeb> It's likely that different compiler will generate different instructions, especially if for some reason they don't get the same flags
[16:24:15] <msanchez> so I was not that off-track. Just wild guessing :)
[16:33:58] <ricea> I think the function you're crashing in comes from here: https://cs.chromium.org/codesearch/f/chromium/src/third_party/skia/src/jumper/SkJumper_generated.S
[16:47:39] <lizeb> So line 684 in your dump maps to _sk_xor__vfp4:
[16:47:47] <lizeb> in the file linked by ricea
[16:48:33] <lizeb> The problem is that the address on which you crash is not an actual instruction address...
[16:49:40] <msanchez> oops! I see
[16:49:52] <msanchez> yeah, it's like in between instructions
[16:50:10] <lizeb> This is not thumb code, so instructions are 4 bytes long
[16:51:15] <msanchez> not sure if it's related, but fwiw this is building with target_cpu="arm" arm_float_abi="hard" arm_use_neon=true arm_use_thumb=true
[16:56:29] <lizeb> I don't think we've ever built chrome on android with GCC 6 as Android doesn't support it, AFAIK. And we now use clang.
--------------

[James Donald (jdonald)]

Thanks Chituc. That's spot-on with the identical error in Firefox. Specifically, it's not an illegal instruction so much as a misaligned (+2 bytes) instruction, plus at that point it's in Thumb mode when anything inside SkJumper should be in ARM mode.

Technically both firefox and chromium-browser are cross-compiled because the Launchpad logs show the host system as aarch64 while the target is armhf:
* https://launchpadlibrarian.net/356307711/buildlog_ubuntu-xenial-armhf.firefox_58.0.2+build1-0ubuntu0.16.04.1_BUILDING.txt.gz
* https://launchpadlibrarian.net/357154172/buildlog_ubuntu-xenial-armhf.chromium-browser_64.0.3282.167-0ubuntu0.16.04.1_BUILDING.txt.gz

Chromium adds a few more flags that might affect NEON i.e. -mfloat-abi=hard -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mthumb, but as you've pointed out the biggest difference is compiling with clang. Apparently clang is actually supported for building Firefox on Linux, so maybe the Launchpad script could be augmented to just add two required lines to mozconfig.

Looks like Firefox 58 for 18.04 Bionic armhf built successfully and uses gcc 7: https://launchpad.net/ubuntu/+source/firefox/58.0.2+build1-0ubuntu1/+build/14326274
Anyone able to try it out?

[Lars Nyqvist (ulos)]

I downloaded firefox_58.0.2+build1-0ubuntu1_armhf.deb and tried to
install but libraries in 17.10 are different
libfontconfig1 (>= 2.12) but 2.11.94-0ubuntu2 ....

On 2/25/18, James Donald <email address hidden> wrote:
> Thanks Chituc. That's spot-on with the identical error in Firefox.
> Specifically, it's not an illegal instruction so much as a misaligned
> (+2 bytes) instruction, plus at that point it's in Thumb mode when
> anything inside SkJumper should be in ARM mode.
>
> Technically both firefox and chromium-browser are cross-compiled because the
> Launchpad logs show the host system as aarch64 while the target is armhf:
> *
> https://launchpadlibrarian.net/356307711/buildlog_ubuntu-xenial-armhf.firefox_58.0.2+build1-0ubuntu0.16.04.1_BUILDING.txt.gz
> *
> https://launchpadlibrarian.net/357154172/buildlog_ubuntu-xenial-armhf.chromium-browser_64.0.3282.167-0ubuntu0.16.04.1_BUILDING.txt.gz
>
> Chromium adds a few more flags that might affect NEON i.e. -mfloat-
> abi=hard -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mthumb, but as you've
> pointed out the biggest difference is compiling with clang. Apparently
> clang is actually supported for building Firefox on Linux, so maybe the
> Launchpad script could be augmented to just add two required lines to
> mozconfig.
>
> Looks like Firefox 58 for 18.04 Bionic armhf built successfully and uses gcc
> 7:
> https://launchpad.net/ubuntu/+source/firefox/58.0.2+build1-0ubuntu1/+build/14326274
> Anyone able to try it out?
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1733340).
> https://bugs.launchpad.net/bugs/1711337
>
> Title:
> Firefox crashes at start on armv7L after 55.0.1 update
>
> Status in Mozilla Firefox:
> Expired
> Status in firefox package in Ubuntu:
> Fix Released
>
> Bug description:
> Firefox always crashes when launched after the 55.0.1 update on an
> Orange Pi PC Plus (a single-board computer similar to a Raspberry Pi),
> even in safe mode.
>
> I did a fresh install of Armbian (a Ubuntu Xenial 16.04 re-spin for
> ARM single-board computer) on a similar board (Orange Pi Plus 2e),
> installed Firefox and experienced the same problem--it won't load
> without crashing.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 16.04
> Package: firefox 55.0.1+build2-0ubuntu0.16.04.2
> Uname: Linux 3.4.113-sun8i armv7l
> AddonCompatCheckDisabled: False
> AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.25.
> ApportVersion: 2.20.1-0ubuntu2.10
> Architecture: armhf
> AudioDevicesInUse:
> USER PID ACCESS COMMAND
> /dev/snd/controlC1: jim 1138 F.... pulseaudio
> /dev/snd/controlC0: jim 1138 F.... pulseaudio
> BuildID: 20170814194718
> Card0.Amixer.info:
> Card hw:0 'audiocodec'/'audiocodec'
> Mixer name : ''
> Components : ''
> Controls : 12
> Simple ctrls : 12
> Card1.Amixer.info:
> Card hw:1 'sndhdmi'/'sndhdmi'
> Mixer name : ''
> Components : ''
> Controls : 1
> Simple ctrls : 1
> Card1.Amixer.values:
> Simple mixer control 'hdmi audio format Function',0
> Capabilities: enum
> Items: '...

[James Donald (jdonald)]

Thanks for trying that, Lars. libfontconfig 2.11 vs 2.12 should be compatible so you could force it to ignore the dependency or run in a sandbox dir without installing. For example: ar xv firefox_58.0.2+build1-0ubuntu1_armhf.deb; tar xvf data.tar.xz; cd ./usr/lib/firefox; ./firefox

This gave me the idea that this could even be tested on Ubuntu MATE 16.04, so I tried that. The only other dependency I had to unpack locally was gcc-7 libstdc++6: https://launchpad.net/ubuntu/bionic/armhf/libstdc++6/7.3.0-1ubuntu1

Unfortunately this gcc-7 build crashes on the same misaligned instruction, so that dashes hopes of resolving this merely with a newer toolchain.

[Chituc Georgian (dianaxxyyzz)]

@James Donald (jdonald) thanks for your reply .All you say is very interesting by my remark is that I think ubuntu do not cross compile firefox for armhf , they just use a aarch64 (arm 64 bit ) kernel and they run a armhf Ubuntu .
I 'm not the expert but from what I talked to others they said it is nto cross compiled and like you said earlier it also say in build logs :
checking whether cross compiling... no
If you can help with what mods are required to make it compile with clang I can try to compile it in a armhf Ubuntu 16.04 .
I'm not a expert, initialy I wanted to try to cross compile it in a x64 intel machine .But I failed to cross compile it for armhf .

[Chituc Georgian (dianaxxyyzz)]

Also I think I'm right it is not cross compiled , you can see at the end of build log :
+------------------------------------------------------------------------------+
| Summary |
+------------------------------------------------------------------------------+

Build Architecture: armhf
Build-Space: 14041372
Build-Time: 24037
Distribution: xenial
Host Architecture: armhf

So the Host Architecture: armhf ....

[Chituc Georgian (dianaxxyyzz)]

My last feeling ,is it just a aarch64 kernel with a arm64 Ubuntu xenial and enabled multiarch so it downloads the armhf build deps . Any way like you said , maybe a build with clang will fix it .
Let;s hope there will be a way to finaly have latest firefox on armhf .
Thank you for your interest to fix this !

[James Donald (jdonald)]

I see. You're right, the kernel is aarch64 but the compiler and build tools are armhf. The chromium-browser build log reports the same, so cross-compilation might not be necessary.

> If you can help with what mods are required to make it compile with clang I can try to compile it in a armhf Ubuntu 16.04 .

According to this link (https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Compiling_Firefox_With_Clang_On_Linux) you just need to edit your mozconfig to set CC=clang and CXX=clang++. In theory you could also test compiler flags like -mfpu=vfpv3-d16 just by setting CFLAGS and CXXFLAGS.

Many thanks if you can try this out in your build environment.

[sam tygier (samtygier)]

armhf build of firefox 58.0.2+build1-0ubuntu1 in ubuntu bionic PROPOSED

Still crashes for me on 18.04 on raspberry pi.

Thread 1 "firefox" received signal SIGILL, Illegal instruction.
0x73b73dfa in _sk_xor__vfp4 () from /usr/lib/firefox/libxul.so
(gdb) bt
#0 0x73b73dfa in _sk_xor__vfp4 () at /usr/lib/firefox/libxul.so
#1 0x73c02484 in SkRasterPipeline::<lambda(size_t, void (* (*)(SkRasterPipeline::StockStage))(), void (*)(), size_t (*)(size_t, void**, K*, size_t))>::operator() (start_pipeline=<optimized out>, just_return=<optimized out>, lookup=<optimized out>, min_stride=2, __closure=<synthetic pointer>)
    at /build/firefox-zKd0bM/firefox-58.0.2+build1/gfx/skia/skia/src/jumper/SkJumper.cpp:320
#2 0x73c02484 in SkRasterPipeline::run_with_jumper(unsigned int, unsigned int) const (this=0x7effa1b8, x=6, n=<optimized out>)
    at /build/firefox-zKd0bM/firefox-58.0.2+build1/gfx/skia/skia/src/jumper/SkJumper.cpp:336
#3 0x0000000c in ()

[Chituc Georgian (dianaxxyyzz)]

Ok , I started to compile it using clang :

 0:55.76 checking for pkg_config... /usr/bin/pkg-config
 0:55.80 checking for pkg-config version... 0.29.1
 0:55.82 checking for yasm... /usr/bin/yasm
 0:55.86 checking yasm version... 1.3.0
 0:55.94 checking the target C compiler version... 3.8.0
 0:56.54 checking the target C compiler works... yes
 0:56.56 checking for the target C++ compiler... /usr/bin/clang++
 0:57.59 checking whether the target C++ compiler can be used... yes
 0:57.59 checking the target C++ compiler version... 3.8.0
 0:58.20 checking the target C++ compiler works... yes
 0:58.22 checking for the host C compiler... /usr/bin/clang
 0:58.73 checking whether the host C compiler can be used... yes
 0:58.73 checking the host C compiler version... 3.8.0
 0:59.34 checking the host C compiler works... yes
 0:59.34 checking for the host C++ compiler... /usr/bin/clang++
 0:59.87 checking whether the host C++ compiler can be used... yes

Will report here if will still crash , in about 20 hours .
It takes a very long time to compile , last time when I compiled it using gcc it was over 20 hours :)
If anybody else have a faster machine and want to try to compile it with clang , based on @James Donald (jdonald) help , I can also help with instruction how to make it start to compiel with clang .
So see ya tomorrow ...

[Chituc Georgian (dianaxxyyzz)]

I confirm they really use clang when they build chromium and they do not cross compile it .
So I have a strong feeling if we compile firefox with clang will work and will be no crash .
My machine is building using clang but will take maybe 1 day to finish.
I see the firefox build for ubuntu devs took max 3 hours to complete .
If somebody with such a machine could try to compile firefox with clang wil'll see faster the result .

[Chris Worsley (c.worsley)]

@ Chituc
Have been googling using clang to compile firefox. This link may be of interest:
https://github.com/JanitorTechnology/dockerfiles/issues/86
see the last post, which indicates support for compiling using clang versions 3.9 through to 6.0.
I see you're using 3.8...

[Chituc Georgian (dianaxxyyzz)]

Thanks @Cris for link , but fortunately my configure step passed and all was fine .
It started to compile is is still compiling for 1:30 hours already .
I did not had that problem ,firefox compile but is very show .I hope will be no errors .
Let's hope building with clang will fix that skia crash

[Chituc Georgian (dianaxxyyzz)]

Firefox looks to dnt get compiled by clang out of the box without intervention.
I got some errors like :

87:09.20 In file included from /root/firefox-58.0.2+build1/media/libpng/arm/filter_neon_intrinsics.c:22:
87:09.22 /usr/lib/llvm-4.0/bin/../lib/clang/4.0.0/include/arm_neon.h:433:1: error: unknown type name 'inline'

And I had to edit firefox-58.0.2+build1/obj-arm-linux-gnueabihf/media/libpng/backend.mk and change "-std=c89" to "-std=gnu89" .
This is just to elp the ones to try to compile firefox with clang

[Chituc Georgian (dianaxxyyzz)]

Can the firefox build be instructed to build only libxul.so ? This will be a faster compile than to build all . I was thinking to build just that with clang and replace this inoriginal deb.

[Chituc Georgian (dianaxxyyzz)]

_sk_xor__vfp4 asm code is thumb2 code so maybe if gcc will be instructed to compile it as a thumb2 code will work also to compile with gcc .

[James Donald (jdonald)]

That's an interesting idea. As I mentioned above chromium-browser builds with -mthumb so it's possible this is a key flag.

Sorry to hear that Firefox does not build with clang out of the box. Another possible approach is to build the majority of Firefox with gcc but the Skia portion in clang. You could take a look at https://hg.mozilla.org/mozilla-central/raw-file/tip/gfx/skia/generate_mozbuild.py and figure out how to override the compiler just for Skia. This would still take at least 20 hours but you are less likely to encounter clang-specific bitrot across Firefox.

To build only libxul.so, as well as to iterate quickly when you attempt fixes like -std=gnu89 to address specific errors, have you tried the instructions at https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Mozilla_build_FAQ#Making_builds_faster ?

Thanks for working on this. It's quite an endeavor to build Firefox from source and we're all cheering you on.

[Chituc Georgian (dianaxxyyzz)]

The good news is that I just had to modify the "-std=" just in 4 places and all compiled fine .
The bad one is that I received a "Memory exausted" error when it build libxul.so .
But I upgraded the machine from 8 gb of RAM to 16 gb of RAM and in 2-3 hours it will try again to build libxul.so .So will see the results soon.

[herrtimson (herrtimson)]

Thank you for your effort! Could you please attach a patch against the firefox source tree for the changes you made, in regards to make it compile with clang?

If you ran out of memory during linking of libxul.so you could try to append '-Wl,--no-keep-memory -Wl,--reduce-memory-overheads' as ldflags, that's what the guys at gentoo are doing at least.

[Chituc Georgian (dianaxxyyzz)]

Here is what I did .

apt-get install build-essential fakeroot devscripts
apt-get build-dep firefox
apt-get install icecc
apt-get install clang-5.0
apt-get install python-dev
apt-get source firefox

cd firefox-58.0.2+build1/

export CC=clang-5.0
export CXX=clang++-5.0
export CCACHE_PREFIX=icecc

dpkg-buildpackage -b

After configure finish and it create stuff in obj-arm-linux-gnueabihf directory , when it started to compile I CTRL+C it .
It generated 3 mozconfig files witch next I eddit to add clang to them too

pico ./debian/config/mozconfig
pico mozconfig
pico browser/config/mozconfig

I added to all this mozconfig files at the top of files :

mk_add_options 'export CCACHE_PREFIX=icecc'
export CC=clang-5.0
export CXX=clang++-5.0

Next all I did was inside obj-arm-linux-gnueabihf directory the dir in witch it compiles all .
So I did not modified the firefox source ,I just adapted the generated backend.mk from obj-arm-linux-gnueabihf and restarted the build process teling him to not clean .
I had to modify :
firefox-58.0.2+build1/obj-arm-linux-gnueabihf/media/libpng/backend.mk and change "-std=c89" to "-std=gnu89"
firefox-58.0.2+build1/obj-arm-linux-gnueabihf/media/ffvpx/libavutil/backend.mk added to the last line -std=gnu99
firefox-58.0.2+build1/obj-arm-linux-gnueabihf/media/ffvpx/libavutil/backend.mk added to the last line -std=c99

Next I restarted the build process teling him to not clean so it did not deleted my modified mozconfig and did not deleted obj-arm-linux-gnueabihf directory , it just startd gain compilation using my modifications .

dpkg-buildpackage -b -nc

If you see any other compile errors expect them to be from incorect "-std=" , mising or wrong one and just modify backend.mk for the libs witch fails in the firefox-58.0.2+build1/obj-arm-linux-gnueabihf/ and restart using "dpkg-buildpackage -b -nc" so it to do not clean your modifications .

[Chituc Georgian (dianaxxyyzz)]

Also in firefox-58.0.2+build1/obj-arm-linux-gnueabihf/gfx/skia/backend.mk on the last line I also have added "-mfloat-abi=hard -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mthumb" hoping to fix the crash in skia . Will see in few hours if will still crash or not ...

[Chituc Georgian (dianaxxyyzz)]

I disasm the SkJumper_generated.o generated by clang : objdump -d SkJumper_generated.o

00000318 <_sk_xor__vfp4>:
     318: f2c70f10 vmov.f32 d16, #1 ; 0x3f800000
     31c: e4913004 ldr r3, [r1], #4
     320: f2603d83 vsub.f32 d19, d16, d3
     324: f2604d87 vsub.f32 d20, d16, d7
     328: f3430d94 vmul.f32 d16, d19, d4
     32c: f3431d95 vmul.f32 d17, d19, d5
     330: f3432d96 vmul.f32 d18, d19, d6
     334: f3433d97 vmul.f32 d19, d19, d7
     338: f2440c90 vfma.f32 d16, d20, d0
     33c: f2441c91 vfma.f32 d17, d20, d1
     340: f2442c92 vfma.f32 d18, d20, d2
     344: f2443c93 vfma.f32 d19, d20, d3
     348: f22001b0 vorr d0, d16, d16
     34c: f22111b1 vorr d1, d17, d17
     350: f22221b2 vorr d2, d18, d18
     354: f22331b3 vorr d3, d19, d19
     358: e12fff13 bx r3

And orriginal SkJumper_generated.S have
HIDDEN _sk_xor__vfp4
.globl _sk_xor__vfp4
_sk_xor__vfp4:
  .long 0xf2c70f10 // vmov.f32 d16, #1
  .long 0xe4913004 // ldr r3, [r1], #4
  .long 0xf2603d83 // vsub.f32 d19, d16, d3
  .long 0xf2604d87 // vsub.f32 d20, d16, d7
  .long 0xf3430d94 // vmul.f32 d16, d19, d4
  .long 0xf3431d95 // vmul.f32 d17, d19, d5
  .long 0xf3432d96 // vmul.f32 d18, d19, d6
  .long 0xf3433d97 // vmul.f32 d19, d19, d7
  .long 0xf2440c90 // vfma.f32 d16, d20, d0
  .long 0xf2441c91 // vfma.f32 d17, d20, d1
  .long 0xf2442c92 // vfma.f32 d18, d20, d2
  .long 0xf2443c93 // vfma.f32 d19, d20, d3
  .long 0xf22001b0 // vorr d0, d16, d16
  .long 0xf22111b1 // vorr d1, d17, d17
  .long 0xf22221b2 // vorr d2, d18, d18
  .long 0xf22331b3 // vorr d3, d19, d19
  .long 0xe12fff13 // bx r3

So it looks the same .I do not have one SkJumper_generated.o build by gcc to compare it with .
Somebody have one ?

[Chituc Georgian (dianaxxyyzz)]

Hmm very interesting . I also objdump -d SkJumper_generated.o ( build by clang without "-mfloat-abi=hard -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mthumb" ) and is different result .
00000318 <_sk_xor__vfp4>:
     318: f2c70f10 .word 0xf2c70f10
     31c: e4913004 .word 0xe4913004
     320: f2603d83 .word 0xf2603d83
     324: f2604d87 .word 0xf2604d87
     328: f3430d94 .word 0xf3430d94
     32c: f3431d95 .word 0xf3431d95
     330: f3432d96 .word 0xf3432d96
     334: f3433d97 .word 0xf3433d97
     338: f2440c90 .word 0xf2440c90
     33c: f2441c91 .word 0xf2441c91
     340: f2442c92 .word 0xf2442c92
     344: f2443c93 .word 0xf2443c93
     348: f22001b0 .word 0xf22001b0
     34c: f22111b1 .word 0xf22111b1
     350: f22221b2 .word 0xf22221b2
     354: f22331b3 .word 0xf22331b3
     358: e12fff13 .word 0xe12fff13

There is the .word ..... objdump result is different .. it is not marked as code ...

[Chituc Georgian (dianaxxyyzz)]

So I heve the strong feeling that here the crash in skia comes from . If "-mfloat-abi=hard -mtune=generic-armv7-a -mfpu=vfpv3-d16 -mthumb" is not passed to clang/gcc , then it does not recognize that part of the binary content as an instruction and it just displays it as if it were a 32-bit data chunk . It show it a a .word not a instruction .

[Chituc Georgian (dianaxxyyzz)]

And that why gdb do not recognize it as a instruction set . We have just to wait few for the compile to finish and I'll tell you if it really works .It is very close to building libxul.so right now .

[Chituc Georgian (dianaxxyyzz)]

When it link libxul.so it get a linker error :

1013:18.63 ../../media/libstagefright/Unified_cpp_media_libstagefright1.o: In function `operator delete(void*)':
1013:18.64 /root/firefox-58.0.2+build1/obj-arm-linux-gnueabihf/dist/include/mozilla/mozalloc.h:230: undefined reference to `free'
1013:18.64 /root/firefox-58.0.2+build1/obj-arm-linux-gnueabihf/dist/include/mozilla/mozalloc.h:230: undefined reference to `free'
1013:18.64 ../../media/libstagefright/Unified_cpp_media_libstagefright1.o: In function `stagefright::AString::clear()':
1013:18.64 /root/firefox-58.0.2+build1/media/libstagefright/frameworks/av/media/libstagefright/foundation/AString.cpp:107: undefined reference to `free'
1013:18.65 /root/firefox-58.0.2+build1/media/libstagefright/frameworks/av/media/libstagefright/foundation/AString.cpp:107: undefined reference to `free'
1013:18.65 ../../media/libstagefright/Unified_cpp_media_libstagefright1.o: In function `operator delete[](void*)':
1013:18.65 /root/firefox-58.0.2+build1/obj-arm-linux-gnueabihf/dist/include/mozilla/mozalloc.h:250: undefined reference to `free'
1013:18.66 ../../media/libstagefright/Unified_cpp_media_libstagefright1.o:/root/firefox-58.0.2+build1/obj-arm-linux-gnueabihf/dist/include/mozilla/mozalloc.h:250: more undefined references to `free' follow
1013:18.66 clang: error: linker command failed with exit code 1 (use -v to see invocation)
1013:18.66 /root/firefox-58.0.2+build1/config/rules.mk:697: recipe for target 'libxul.so' failed
1013:18.67 make[6]: *** [libxul.so] Error 1
1013:18.67 make[6]: Leaving directory '/root/firefox-58.0.2+build1/obj-arm-linux-gnueabihf/toolkit/library'
1013:18.67 /root/firefox-58.0.2+build1/config/recurse.mk:73: recipe for target 'toolkit/library/target' failed

Any ideea about what ldflags must be set on libstagefright so it to find the free() ?

[Chituc Georgian (dianaxxyyzz)]

I can not pass pass about this linking problem , because re running the build process is very time consuming . Any way I will let it build with gcc and I will replace in gcc build the gfx/skia dir with the one build by clang .

[Chris Worsley (c.worsley)]

@Chituc
I don't know if this is a helpful idea, or not, but here goes.
The problems are:
1 knowledge/expertise re compiling and linking specific to Firefox (sadly I can't help there - no experience)
2 Long compile times to iterate - you've asked if anyone has arm64 that could help.

Re 2, would using arm64 in the cloud help?
https://www.scaleway.com/ appears to offer an ARM64 server with 8 armv8 cores, 8GB memory, 200GB ssd on a pay-as-you-go basis (12euro a month).
They have ubuntu images, but I'd expect it would require uploading your armhf build environment & some configuring..
Not sure how much RAM you actually need, but they appear to have larger RAM configurations, with more cores, at higher cost of course.
Looking at their faq/servers page, it looks like the default ubuntu image is Ubuntu 14.04 LTS, but they also offer Ubuntu LTS 16.04 (Xenial) & Ubuntu 17.04 (Zesti)

Performance isn't guaranteed though, as this is shared infrastructure, which is probably really designed for web services..

If this looks like it would make the difference in iteration times, I'd be happy to contribute.

[Chituc Georgian (dianaxxyyzz)]

Thanks Chris that is hopeful to know .
As a summary for all I did and some notes :

Compiling firefox with clang was fine ,just required some -std=xxx adaption.
But after it finish to compile it try to make libxul.so by linking libs and it fail to link libstagefright teling me it can not find 'free()' .

The clang sucesufly compiled SkJumper_generated.o looks different from objdump perspective than the SkJumper_generated.o compiled without some flags ,so I think the one compiled by clang with flags will not crash firefox .

I started again the build of firefox but this time I make a untouched build ,I just let gcc/g++ to compile and link all ,same like ubuntu devs do , just to see all compile and link good.
It require 4-5 hours more to finish .

After I'll see all compiled good with gcc , I will just replace the SkJumper_generated.o build by gcc with the one build by clang ,and I'll put it to make again the libxul.so ,and test if any crash .

So for now I just wait to see it finish compiling with gcc same like ubuntu devs do , so next me to change SkJumper_generated.o .

[Chituc Georgian (dianaxxyyzz)]

It successfully build a deb .I build all with gcc except gfx/skia build with clang .
There is again a crash but It looks now is not in _sk_xor__vfp4 the asm code where it crash is different from the asm code of _sk_xor__vfp4 .

firefox -g
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/firefox/firefox...(no debugging symbols found)...done.
(gdb) run
Starting program: /usr/lib/firefox/firefox
Cannot parse expression `.L1185 4@r4'.
warning: Probes-based dynamic linker interface failed.
Reverting to original interface.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[New Thread 0xf2522450 (LWP 2431)]
[New Thread 0xf1d22450 (LWP 2432)]
[Thread 0xf1d22450 (LWP 2432) exited]
[New Thread 0xf1d22450 (LWP 2433)]
[New Thread 0xed17b450 (LWP 2434)]
[New Thread 0xec97b450 (LWP 2435)]
[New Thread 0xec17b450 (LWP 2436)]
[New Thread 0xf12ff450 (LWP 2437)]
[New Thread 0xf10ff450 (LWP 2438)]
[New Thread 0xeb97b450 (LWP 2439)]
[New Thread 0xeb77b450 (LWP 2440)]
[New Thread 0xeb57b450 (LWP 2441)]
[New Thread 0xeb37b450 (LWP 2442)]
[New Thread 0xeb17b450 (LWP 2443)]
Thread 1 "firefox" received signal SIGSEGV, Segmentation fault.
0xf4c3dcb4 in ?? () from /usr/lib/firefox/libxul.so
(gdb) disas /r 0xf4c3dca0,0xf4c3dcd0
Dump of assembler code from 0xf4c3dca0 to 0xf4c3dcd0:
   0xf4c3dca0: 9b 68 ldr r3, [r3, #8]
   0xf4c3dca2: 05 eb 83 03 add.w r3, r5, r3, lsl #2
   0xf4c3dca6: 16 46 mov r6, r2
   0xf4c3dca8: 0f 46 mov r7, r1
   0xf4c3dcaa: 98 44 add r8, r3
   0xf4c3dcac: 20 99 ldr r1, [sp, #128] ; 0x80
   0xf4c3dcae: d9 e9 00 23 ldrd r2, r3, [r9]
   0xf4c3dcb2: 82 46 mov r10, r0
=> 0xf4c3dcb4: c1 e9 00 23 strd r2, r3, [r1]
   0xf4c3dcb8: d9 f8 04 30 ldr.w r3, [r9, #4]
   0xf4c3dcbc: 7b 33 adds r3, #123 ; 0x7b
   0xf4c3dcbe: 00 f0 4b 81 beq.w 0xf4c3df58
   0xf4c3dcc2: b1 69 ldr r1, [r6, #24]
   0xf4c3dcc4: c1 f3 56 25 ubfx r5, r1, #9, #23
   0xf4c3dcc8: 21 f4 ff 71 bic.w r1, r1, #510 ; 0x1fe
   0xf4c3dccc: 21 f0 01 01 bic.w r1, r1, #1

I dnt know where this asm code belongs too . I dnt know in what function it crash , but is not in sk jumper generated .

[Chituc Georgian (dianaxxyyzz)]

I installed debug symbols here are more crash about the crash , again like i said is not in skia anymore :

Thread 1 "firefox" received signal SIGSEGV, Segmentation fault.
0xf4c3dcb4 in JS::MutableHandle<JS::Value>::set (v=..., this=<synthetic pointer>) at /root/firefox-58.0.2+build1/obj-arm-linux-gnueabihf/dist/include/js/RootingAPI.h:580
580 /root/firefox-58.0.2+build1/obj-arm-linux-gnueabihf/dist/include/js/RootingAPI.h: Permission denied.
(gdb)

[Chituc Georgian (dianaxxyyzz)]

Maybe because it do not crash on skia anymore , this crash in RootingAPI.h will be easier to fix ...

[James Donald (jdonald)]

Nice work!

If you edit your prefs.js to disable JIT (javascript.options.baselinejit = false) or try to disable JavaScript altogether (java.script.enabled = false), does it crash at the same point?

(I would expect so as these flags have not helped much when troubleshooting in the past, but it's still something I check whenever seeing a backtrace with JS::)

[Chituc Georgian (dianaxxyyzz)]

I tried yesterday java.script.enabled false in prefs.js and it still crashed , same crash .
i will try also javascript.options.baselinejit false ;

You know firefox 57.04 armhf , did not used sk jumper generated code with skia disabled in prefs.js and worked fine .

I have to see if there was any change in code of RootingAPI.h from firefox 57 to 58 .

I also want to build firefox 57.04 armhf with the sk jumper generated compiled by clang and see if it works without to disable skia in prefs.js .

The only problem is it takes a lot to compile in my machine .

[Chituc Georgian (dianaxxyyzz)]

@james you said in a old post

"You can actually hexedit libxul.so and change those four vorr instructions into NOPs. Firefox then makes it past this point but crashes obscurely elsewhere, with different crash signatures on Firefox 58 vs 59."

Can you check where that "crashes obscurely elsewhere" crashed ? Maybe it crashed exacly where it crashed for me ? I think skia crash was resolved by building skia with clang , but I'm curios if for you the crash was same JS crash like I had

[James Donald (jdonald)]

There was a typo in my above post, should say javascript.enabled, but I'll assume you didn't have the extra dot when tried yesterday.

> I installed debug symbols

I just realized this makes it sound like you installed firefox-dbg, which would give incorrect addresses because those symbols are for the public apt binary. Do you mean that you *rebuilt* with debug symbols (-g)?

> Can you check where that "crashes obscurely elsewhere" crashed ?

My NOP test earlier was flawed because it did not get the program counter back onto a 4-byte aligned instruction + switch out of Thumb mode. The fix was to hexedit in the thumb instructions 0020; 0047 (movs r0, #0; bx r0). With this, Firefox 58 continues further on and then crashes here:

   0x0040517c <_start+0>: 4f f0 00 0b mov.w r11, #0
   0x00405180 <_start+4>: 4f f0 00 0e mov.w lr, #0
   0x00405184 <_start+8>: 02 bc pop {r1}
   0x00405186 <_start+10>: 6a 46 mov r2, sp
   0x00405188 <_start+12>: 04 b4 push {r2}
   0x0040518a <_start+14>: 01 b4 push {r0}
   0x0040518c <_start+16>: df f8 24 a0 ldr.w r10, [pc, #36] ; 0x4051b4 <_start+56

Firefox 59 crashes at the same instruction sequence, although the addresses are of course different.

Getting a backtrace with symbols: Unfortunately for me if I install firefox-dbg, firefox -g just hangs indefinitely. I thought this was due to the limit of 1 GB RAM on a Raspberry Pi 3, but it doesn't get any further even if I add gigabytes of swap space or wait a while.

It's also possible that the crash you're hitting is in the neighborhood of the other one we were avoiding on Firefox 57 by specifically installing the 14.04 package. The 16.04 packages of 57, 58, and 59 don't even hit the SkJumper bug, instead crashing at an earlier point on this:

=> 0xf314e830: c1 e9 00 23 strd r2, r3, [r1]
   0xf314e834: 05 9b ldr r3, [sp, #20]
   0xf314e836: 06 9a ldr r2, [sp, #24]
   0xf314e838: 1a 60 str r2, [r3, #0]
   0xf314e83a: 9d f8 38 30 ldrb.w r3, [sp, #56] ; 0x38

Although this does not appear the same as the crash on your custom build, interestingly in both cases the segfaulting instruction is precisely strd r2, r3, [r1]

To confirm if you're actually getting past the SkJumper problem point, after a few seconds the Firefox window frame should appear on the screen before it crashes. That's the behavior on firefox_58.0+build6-0ubuntu0.14.04.1_armhf.deb currently while firefox_58.0.2+build1_0ubuntu0.16.04.1_armhf.deb just aborts before getting that far.

[Chituc Georgian (dianaxxyyzz)]

I will install default ubuntu xenial firefox_58.0.2 and see if that crash on a different asm instruction than my custom build and I will report .
By the way I managed to set up a cross compile development , now I compile it with a intel cpu and is a lot faster . So a lot of testing can be made . You sugest me to get the firefox 58 from ubuntu 14 ?

[Chituc Georgian (dianaxxyyzz)]

I run firefox 58.02 armhf that comes with ubuntu 16.04 and it crash in same point my custom firefox 58.02 crashed :

Thread 1 "firefox" received signal SIGSEGV, Segmentation fault.
0xf4c3bfc4 in JS::MutableHandle<JS::Value>::set (v=..., this=<synthetic pointer>)
    at /build/firefox-ID1dFf/firefox-58.0.2+build1/obj-arm-linux-gnueabihf/dist/include/js/RootingAPI.h:580
580 /build/firefox-ID1dFf/firefox-58.0.2+build1/obj-arm-linux-gnueabihf/dist/include/js/RootingAPI.h: No such file or directory.
(gdb) diass /r 0xf4c3bfc0,0xf4c3bfd0
Undefined command: "diass". Try "help".
(gdb) disas /r 0xf4c3bfc0,0xf4c3bfd0
Dump of assembler code from 0xf4c3bfc0 to 0xf4c3bfd0:
   0xf4c3bfc0 <js::jit::DoTypeMonitorFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICTypeMonitor_Fallback*, JS::HandleValue, JS::MutableHandleValue)+96>: 00 23 movs r3, #0
   0xf4c3bfc2 <js::jit::DoTypeMonitorFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICTypeMonitor_Fallback*, JS::HandleValue, JS::MutableHandleValue)+98>: 82 46 mov r10, r0
=> 0xf4c3bfc4 <js::jit::DoTypeMonitorFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICTypeMonitor_Fallback*, JS::HandleValue, JS::MutableHandleValue)+100>: c1 e9 00 23 strd r2, r3, [r1]
   0xf4c3bfc8 <js::jit::DoTypeMonitorFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICTypeMonitor_Fallback*, JS::HandleValue, JS::MutableHandleValue)+104>: d9 f8 04 30 ldr.w r3, [r9, #4]
   0xf4c3bfcc <js::jit::DoTypeMonitorFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICTypeMonitor_Fallback*, JS::HandleValue, JS::MutableHandleValue)+108>: 7b 33 adds r3, #123 ; 0x7b
   0xf4c3bfce <js::jit::DoTypeMonitorFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICTypeMonitor_Fallback*, JS::HandleValue, JS::MutableHandleValue)+110>: 00 f0 4b 81 beq.w 0xf4c3c268 <js::jit::DoTypeMonitorFallback(JSContext*, js::jit::BaselineFrame*, js::jit::ICTypeMonitor_Fallback*, JS::HandleValue, JS::MutableHandleValue)+776>
End of assembler dump.
(gdb)

The problem is same RootingAPI.h:580 . Ok so in firefox 58.02 we have 2 bugs , this one "RootingAPI.h:580" and the _sk_xor__vfp4 that I hope to get fixed by compiling with clang .

I will get the Firefox 57.04 source from Ubuntu 14 and check if it still crash if I rebuild it with Skjumper build with clang .I want to see if rebuilding SkJumpr generated with clang fix the _sk_xor__vfp4 crash .

[James Donald (jdonald)]

> By the way I managed to set up a cross compile development

That's great news. Could you post your instructions? This would allow others here to help out, considering most of us don't have ARM boards with much RAM or 20 hours of patience.

This also opens up the possibility of making a build on Debian 9 to match up better with Raspbian Stretch, rather than having to resort to 14.04 binaries there.

> Ok so in firefox 58.02 we have 2 bugs , this one "RootingAPI.h:580" and the _sk_xor__vfp4 that I hope to get fixed by compiling with clang .

It appears that way, although worth noting that the crash at startup also happens in Firefox 57, specific to the 16.04 build. You can check that out by doing wget http://launchpadlibrarian.net/352606050/firefox_57.0.4+build1-0ubuntu0.16.04.1_armhf.deb and running it.

> You sugest me to get the firefox 58 from ubuntu 14 ?

Yeah, it appears to get around the crash at startup (before any window appears, and thus before the SkJumper part) it might be necessary to build under Ubuntu 14.04 toolchains (gcc 4.8, libc 2.19).

[Chituc Georgian (dianaxxyyzz)]

 firefox_57.0.4 ubuntu xenial crash like firefox 58 ubuntu xenial , before it reach skia bug
=> 0xf4cd4790: strd r2, r3, [r1]
   0xf4cd4794: ldr.w r3, [r9, #4]
   0xf4cd4798: adds r3, #123 ; 0x7b
   0xf4cd479a: beq.w 0xf4cd4a34
   0xf4cd479e: ldr r1, [r6, #24]

 firefox_57.0.4 ubuntu trusty crash at skia bug
=> 0xf45d4b32: 20 f2 b1 11 ; <UNDEFINED> instruction: 0xf22011b1
   0xf45d4b36: 21 f2 b2 21 ; <UNDEFINED> instruction: 0xf22121b2
   0xf45d4b3a: 22 f2 b3 31 ; <UNDEFINED> instruction: 0xf22231b3
   0xf45d4b3e: 23 f2 13 ff bl 0xf47f8968

So firefox build with xenial toolchain have one more crash , mostly because of different build enviroment from trusty to xenial .

So a solution will be to build firefox with trusty toolchain .

About the cross compile set up , I was happy but to early , it compile fine for 30 minutes then stop compilation because a bug in rustc compiler .
I most used https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Build_Instructions/Cross-compiling_Mozilla and set up a xenial chroot . I installed some packages for xenial some for armhf . I uset rustup to install rust that can make target armhf .
But at one point I think it mix the include headers of x86 with the ones from arm .

I think the best is liek you said to set up a arm machine running ubuntu trusty and use that to compile all . I think that way will work . Using arm xenial just give one (or more ) crash not just the skia one .

[herrtimson (herrtimson)]

Which is the bug id for the mentioned skia bug?

[James Donald (jdonald)]

I think we just have the IRC logs that Chituc posted, no bug id.

When I searched https://bugs.chromium.org/p/skia/issues/list last months for issues with the keyword "armhf" I did not find any such bug filed.

At the time I was about to file a ticket myself, but that made me first run through the thought process of what their initial response might be. Imagining that to be "well, it works for Chromium" is what inspired me to look more closely at the chromium-browser build logs.

[Lars Nyqvist (ulos)]

Firefox 59 fails as earlier .

Mozilla Crash Reporter:

Add-ons: activity-stream%40mozilla.org:2018.02.17.0026-173e2795,aushelper%40mozilla.org:2.0,firefox%40getpocket.com:1.0.5,followonsearch%40mozilla.com:0.9.6,formautofill%40mozilla.org:1.0,onboarding%40mozilla.org:1.0,screenshots%40mozilla.org:25.0.0,shield-recipe-client%40mozilla.org:80,webcompat%40mozilla.org:1.1,%7B972ce4c6-7e08-4474-a285-3208198ce6fd%7D:59.0,langpack-fi%40firefox.mozilla.org:59.0,langpack-en-GB%40firefox.mozilla.org:59.0,langpack-en-ZA%40firefox.mozilla.org:59.0
BuildID: 20180313132558
CrashTime: 1521057567
DOMIPCEnabled: 1
EMCheckCompatibility: true
Email:
FramePoisonBase: 0000004041121792
FramePoisonSize: 4096
InstallTime: 1521056803
Notes: Ubuntu 17.10FP(D00-L1000-W00000000-T000) OpenGL: VMware, Inc.
-- llvmpipe (LLVM 5.0, 128 bits) -- 3.0 Mesa 17.2.8 --
texture_from_pixmap
WR? WR- OMTP? OMTP-
ProductID: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
ProductName: Firefox
ReleaseChannel: release
SafeMode: 0
SecondsSinceLastCrash: 754
StartupCrash: 0
StartupTime: 1521057071
TelemetryEnvironment:
{"build":{"applicationId":"{ec8030f7-c20a-464f-9b0e-13a3a9e97384}","applicationName":"Firefox","architecture":"arm","buildId":"20180313132558","version":"59.0","vendor":"Mozilla","platformVersion":"59.0","xpcomAbi":"arm-eabi-gcc3","hotfixVersion":null,"updaterAvailable":false},"partner":{"distributionId":null,"distributionVersion":null,"partnerId":null,"distributor":null,"distributorChannel":null,"partnerNames":[]},"system":{"memoryMB":2014,"virtualMaxMB":null,"cpu":{"count":4,"cores":null,"vendor":null,"family":null,"model":null,"stepping":null,"l2cacheKB":null,"l3cacheKB":null,"speedMHz":1608,"extensions":["hasEDSP","hasARMv6","hasARMv7","hasNEON"]},"os":{"name":"Linux","version":"4.13.5","locale":"fi-FI"},"hdd":{"profile":{"model":null,"revision":null},"binary":{"model":null,"revision":null},"system":{"model":null,"revision":null}},"gfx":{"D2DEnabled":null,"DWriteEnabled":null,"ContentBackend":"Cairo","adapters":[{"description":"VMware,
Inc. -- llvmpipe (LLVM 5.0, 128 bits)","vendorID":"VMware,
Inc.","deviceID":"llvmpipe (LLVM 5.0, 128
bits)","subsysID":null,"RAM":null,"driver":null,"driverVersion":"3.0
Mesa 17.2.8","driverDate":null,"GPUActive":true}],"monitors":[],"features":{"compositor":"none","gpuProcess":{"status":"unused"}}},"appleModelId":null},"settings":{"blocklistEnabled":true,"e10sEnabled":true,"e10sMultiProcesses":4,"telemetryEnabled":false,"locale":"en-US","update":{"channel":"release","enabled":false,"autoDownload":true},"userPrefs":{"browser.search.widget.inNavBar":false,"browser.startup.homepage":"<user-set>","general.config.filename":"<set>"},"sandbox":{"effectiveContentProcessLevel":null},"addonCompatibilityCheckEnabled":true,"isDefaultBrowser":null},"profile":{}}
Theme: classic/1.0
ThreadIdNameMapping: 12238:"Gecko_IOThread",12240:"Timer",12241:"Link
Monitor",12242:"Socket Thread",12243:"JS
Watchdog",12252:"BGReadURLs",12253:"Hang Monitor",12262:"Cache2
I/O",12263:"Cookie",12266:"DOM Worker",12267:"IPDL
Background",12271:"LoadRoots",12273:"GMPThread",12274:"SoftwareVsyncThread",12275:"Compositor",12276:"VRListener",12277:"ImgDecoder
#1",12278:"ImgDecoder #2",12...

[herrtimson (herrtimson)]

bug still not solved, got the update to firefox-59 on lubuntu-14.04 today, it still segfaults. So I spent a few hours to an install of arch linux on my rpi2, simply because firefox never had this issue over there, and it still doesn't hit this bug as of today.

arch linux is rolling release, atm the toolchain is: gcc-7.2.1, glibc-2.26, binutils-2.29 or 2.30, rust-1.24.1, clang/llvm-5.0.1

I made a screenshot for everyone, with the cflags and compile switches. --enable-rust-simd caught eye, what is this all about? Also anyone tried to run the binary from bionic, which should be built with gcc-7 as well?

[James Donald (jdonald)]

Awesome, thanks for confirming Arch still works and posting the screenshot.

> Also anyone tried to run the binary from bionic, which should be built with gcc-7 as well?

Yes, Lars and I both tried this. It's rather buried in discussion above but search for the part where I mention "ignore the dependency" in order to test the gcc 7 Bionic build on a Xenial system.

> --enable-rust-simd caught eye, what is this all about?

There are Rust bindings for Skia, but the part where it crashes doesn't use Rust. From your screenshot I think these are likely the key missing flags: -march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16

In the conversation above, those are exactly the flags we suspected from comparing the Chromium build. Looks like we don't actually need clang, and we don't need -mthumb

Well, there's still the other startup crash specific to Ubuntu 16.04 packages. Given we don't hit it on the Bionic nor Trusty builds, that problem is likely specific to gcc 5.4

Now we just need someone with access to change the default build settings on Launchpad.

[Chituc Georgian (dianaxxyyzz)]

Can you check if the firefox 57.04 trusty , the working one shows "-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" in about:buildconfig ?

I'm thinking that "-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" are automaticaly added when you build something with gcc in Ubuntu armhf .

[Chris Worsley (c.worsley)]

@Chituc:
I’m still running that build from my Debian aid environment so just checked now.
It’s not showing those flags in about:buildconfig

[Chituc Georgian (dianaxxyyzz)]

Ok , I did not tried firefox 59 but my firefox 58.02armhf build with this options "-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" was compiled fine by gcc under Ubuntu 16.04 and there was still a crash .

You can read the crah cause I post earlier . Is not the skia crash , is another one that happens before it reach skia .
So just compiling with "-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" may not be sufficient, at lest for me was not .

[Chituc Georgian (dianaxxyyzz)]

But there is a big hope the build for trusty with "-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" will not crash . Builing on xenial did nto worked but a build in trusty have great chances to work .

[herrtimson (herrtimson)]

@Chituc #92 : don't think I can provide about:buildconfig from 57.x , because they segfault. If there's a chance to get the output of about:buildconfig regardless the segfault at startup, I'm all eager to learn about this.

On trusty, one can see the browser window maybe for 1/3 of a second or so, before it crashes. This wasn't the case before, so it seems to go into the right direction here! ;-) I'm all optimistic we can sort this out, even more since it has proven to be possible on Arch linux.

Why do you think that setting cflags as in CFLAGS="-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" would make it not segfault?

[Chituc Georgian (dianaxxyyzz)]

We think CFLAGS="-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" will prevent the skia crash .
But firefox must be build using trusty cause xenial builds have at least 2 crashes . (skia and smth else) .

The trusty build crash just if skia is enabled and we think building firefox using trusty and CFLAGS="-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" will be just fine

[Chituc Georgian (dianaxxyyzz)]

If you have a good intel cpu and 16 gb ram ,you can just install using sbuild the trusty armhf and use it to compile . It will take no more than 3 days to compile on a i7 intel cpu .
It will take longer than it took inside my real arm hardware because it use qemu , but will not take more than 3 days to compile , so if you have a extra pc you can use sbuild to set up trusty armhf and CFLAGS="-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" .

[herrtimson (herrtimson)]

Hmm, why don't you poke the person who's responsible for the packaging with this, to adjust the build script in a matter of passing different cflags if using armv7 and trusty? I'm not really keen on setting up a specific chroot for trusty, mostly because I have a limited internet connection. The compile is not that much of a burden, for 59.0 it takes 10 hours on a rpi2 with -j2 and a bit of swapping, as non debug build. Can be done overnight.

Clang is more memory efficent, so you can pass -j4 on the rpi2 quadcore and swap a bit here and there, the built would be a little bit faster this way and might allow to workaround the segfault. But again Ubuntu doesn't have >=clang-4 for armhf and trusty available, does it?

[Chituc Georgian (dianaxxyyzz)]

I never talked with a ubuntu package manager and I do not know anybody ... I have a HTC 10 phone with 4 gb ram that I can use to try to build firefox on it . I did not used it till now cacse I use it without swap and firefox team say we need at least 8 gb ram to compile ff . Are you sure 2 gb ram and a lot of swap is fine to compile ? Maybe somebody can contact ff package manager and tell him to try to compile on trusty with CFLAGS="-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" . I see in the build logs for them took just 2 hours and half .

[herrtimson (herrtimson)]

I wrote an email to Chris Coulson, asking for some assistance with the cflags. Hopefully he'll read it.

What you're saying about hardware requirements is not really true. Firefox is pretty demanding in terms of memory, yes, but per job mostly. On amd64 and if you build with gcc, it needs roughly 700 to 800 mb of ram per thread, and roughly double as much if you pass debug cflags along. Thats as bad as it gets, the demand in ram decreases a lot if you take clang instead, which is also a little bit faster and the binary seems to be a little bit faster as well (I'm actually writing this post from a clang built firefox) However, situation on armhf is more tense, because if you aim to compile nativly they are often limited by the amount of ram per job. rpi2 has only ~900mb free available at boot time, so we end up with just 200mb per core. This is not enough if you use gcc for compilation, the device would swap a lot, whereas with with -j2 and 400mb per thread it only swaps during some hefty code sections around the java script engine and close to the finish of the build, during linking.

Swapping can be so slow, that it completly eats away the theoretical speed gain by allowing more cores to be used with make via -jN

Also I would like to mention that we propably have to stick to gcc for compilation, because you can use clang to compile firefox on amd64, but not on armhf, due to a linking error. I think it is the same as in #72 of this bug report:

../../media/libstagefright/Unified_cpp_media_libstagefright1.o: In function `stagefright::SampleIterator::getChunkOffset(unsigned int, long long*)':
../../media/libstagefright/Unified_cpp_media_libstagefright1.cpp:(.text+0x554): undefined reference to `free'
../../media/libstagefright/Unified_cpp_media_libstagefright1.o: In function `stagefright::SampleIterator::getSampleSizeDirect(unsigned int, unsigned int*)'

At the moment trying to use gold linker instead of bfd, we'll see if that makes a difference.

[herrtimson (herrtimson)]

there is one bug at mozilla about clang and arm, but it is more about android, I guess. Still: https://bug635044.bugzilla.mozilla.org/show_bug.cgi?id=1163171

Also freebsd builds with clang by default, they have a bug with a patch to fix it at their end for armv7, it might be possible to adapt this patch for ubuntu, if we want to switch over to build firefox with clang on arm.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=225279

[herrtimson (herrtimson)]

This is the freebsd patch from their bug report. It was generated against their ports tree, I edited it so that it can be applied against the firefox source tree instead. This is not an official patch, and it fixes freebsd only, but still it might be helpfull to see what they changed. It is confirmed to build at their ends with clang for armv6 + armv7

adding this just in case it might be helpfull at some point

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "firefox-esr-arm-clang-FREEBSD.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[James Donald (jdonald)]

> [herrtimson] I wrote an email to Chris Coulson, asking for some assistance with the cflags. Hopefully he'll read it.

Thanks. I flagged him on #ubuntu-devel IRC as well. Hopefully we'll get his attention this time.

> > [Chituc] Can you check if the firefox 57.04 trusty , the working one shows
"-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" in about:buildconfig ?
> [Chris] It’s not showing those flags in about:buildconfig

I don't think this matters, because Firefox 57 Trusty still has the Skia crash for most armhf users. The main difference between Firefox 57 and 58/59 is that in Firefox 57 it's possible to disable Skia via prefs.js, but otherwise the misaligned instruction bug is still there.

> it fixes freebsd only, but still it might be helpfull to see what they changed.

The majority of that patch deals with atomics, compare-and-swap, and memory barriers, although might be worth noticing that it does add -march=armv7-a

> [Chituc] Builing on xenial did nto worked but a build in trusty have great chances to work .

Right, but important to note that building on Bionic will likely work too. That point may be helpful in getting the Ubuntu team to prioritize.

> CFLAGS="-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16"

In case anyone is testing this soon, be sure to set both CFLAGS and CXXFLAGS. SkJumper_generated.S is assembly and I'm not sure which of the two flag sets gets passed, but the Skia project is largely C++.

[Adam Conrad (adconrad)]

CFLAGS="-march=armv7-a -mfloat-abi=hard -mfpu=vfpv3-d16" is the default for armhf. In fact, it's what *defines* armhf as armhf. Unless something is explicitly setting those things differently, they don't need to be on the commandline at all.

[James Donald (jdonald)]

Ugh, yeah doing some simple helloworld tests confirms these are armhf gcc *defaults*.

> it's what *defines* armhf as armhf.

Defines: This appears true for -march=armv7-a -mfloat-abi=hard. As for -mfpu, adding -mfpu=neon results in some assembly differences, namely using register d16 and above.

> Unless something is explicitly setting those things differently

The Firefox build log has some instances of -mfpu=neon, particularly in Skia. However, Chromium does as well so indeed this is a longshot.

Looks like we're back to the drawing board. Still some hope with changes like building Skia using clang, adding -mthumb, or -fstack-protector-strong, but none of these other differences stand out as something present in both the Chromium and Arch Linux builds.

I think I misunderstood earlier about the intent of looking at Firefox 57's about:buildconfig. As that's technically still a Skia-broken build, we can continue to examine that configuration for other differences vs the Arch Linux one.

> [herrtimson] If there's a chance to get the output of about:buildconfig regardless the segfault at startup, I'm all eager to learn about this.

With broken browsers like Firefox 58 and 59 armhf, find the build config like so:

strings /usr/lib/firefox/omni.ja | grep -A 31 "Built from"

[James Donald (jdonald)]

@Chituc,

I attempted to set up an Ubuntu 14.04 Firefox build environment, and am realizing how difficult it is to troubleshoot something old and barely supported.

Instead, could you take your existing 16.04 build flow but run sudo apt install g++-4.8 then build with CC=gcc-4.8, CXX=g++-4.8?

There's a decent chance this will get you past the strd r2, r3, [r1] crash, which would then allow you to complete your test of the clang Skia theory.

[herrtimson (herrtimson)]

Ubuntu builds firefox for trusty with g++-4.9 , I checked this on amd64 some time ago and the build logs tell me that the armhf toolchain is as well g++-4.9

 0:08.79 checking for the target C++ compiler... /<<BUILDDIR>>/firefox-59.0.1+build1/debian/gcc-mozilla/g++
 0:08.90 checking whether the target C++ compiler can be used... yes
 0:08.90 checking the target C++ compiler version... 4.9.4
 0:08.98 checking the target C++ compiler works... yes
 0:08.98 checking for the host C compiler... /<<BUILDDIR>>/firefox-59.0.1+build1/debian/gcc-mozilla/gcc
 0:09.04 checking whether the host C compiler can be used... yes
 0:09.04 checking the host C compiler version... 4.9.4
 0:09.11 checking the host C compiler works... yes
 0:09.11 checking for the host C++ compiler... /<<BUILDDIR>>/firefox-59.0.1+build1/debian/gcc-mozilla/g++
 0:09.18 checking whether the host C++ compiler can be used... yes
 0:09.18 checking the host C++ compiler version... 4.9.4
 0:09.25 checking the host C++ compiler works... yes

[herrtimson (herrtimson)]

If I use gcc-6.4.0 to compile with -g I get the same corrupt stack error as @levente before. Currently thinking about upgrading to gcc-7, as a last ressort, or waiting for bionic release in roughly a month or so.

[James Donald (jdonald)]

@herrtimson we already confirmed that the gcc-7 / Bionic build crashes on the exact same misaligned instruction in Skia. See responses #48 and #50 for how I tested this, the point being that waiting for Bionic alone won't fix this.

Diagnosing a corrupt stack: whether you have debug symbols or not you should run this in gdb: layout asm. This will tell you if it's the same Skia illegal/misaligned instruction, the earlier SIGSEGV crash at strd r2, r3, [r1], or something else. Because levente was using a 16.04 deb he was most likely hitting the known SIGSEGV. Would be good to identify your build's crash because we don't have any other datapoints for gcc 6.

[Chris Worsley (c.worsley)]

I thought it might be useful for you folks who have compiling experience to compare the compiler switches used in the Trusty build for 57.04 with those from Arch 59.0 (from post #90), so I've listed them in columns (attached), after I attempted to pull the switches out of @herrtmison's screenshot (might be a few errors from that though).

There are lots more switches in the Arch version... Perhaps a look at what's different may spark some ideas..

[herrtimson (herrtimson)]

@jdonald: how can I direct the output of 'layout asm' from gdb into a textfile?

[James Donald (jdonald)]

Thanks @Chris!

@herrtimson you can do this:
  layout asm
  tui disable
  set logging on
  disas /r <starting address>,<ending address>

It'll save the output of your disas command to gdb.txt

[James Donald (jdonald)]

@levente (and maybe @herrtimson),

I set up a new Ubuntu MATE system, and in the process realized that your segfault in /lib/ld-linux-armhf.so.3 is a symptom of bug 1576432. That's where gdb fails not only for Firefox but even a simple helloworld program. The workaround there is to install libc6-dbg, which will allow you to see one of the other two crash signatures that we still don't have a solution for.

@Chituc,

By waiting a couple days on my Pi 3 I managed to make a gcc-4.9 build of Firefox 59.0.1, which then crashes on strd r2, r3, [r1]. I'm now thinking that this startup crash may be more nuanced than being specific to Ubuntu 16.04 or gcc-5.4. In fact, the Launchpad builds are now showing something odd: While the 59.0.1 14.04 build that we've talked about gets all the way to the Skia SIGILL, the recent 59.0.2 14.04 build from https://launchpad.net/ubuntu/trusty/armhf/firefox/59.0.2+build1-0ubuntu0.14.04.1 segfaults on the strd too.

At least with a custom build I'm now able to get accurate debug symbols. With the TUI I inspected the source at each level of this stack trace:

#0 0x717c4bc8 in JS::Value::setObjectNoCheck (obj=0x67d62160, this=<optimized out>) at /home/jdonald/firefox-59.0.1+build1/obj-arm-linux-gnueabihf/dist/include/js/Value.h:380
#1 JS::Value::setObject (obj=..., this=<optimized out>) at /home/jdonald/firefox-59.0.1+build1/obj-arm-linux-gnueabihf/dist/include/js/Value.h:375
#2 js::MutableWrappedPtrOperations<JS::Value, JS::MutableHandle<JS::Value> >::setObject (obj=..., this=<synthetic pointer>)
    at /home/jdonald/firefox-59.0.1+build1/obj-arm-linux-gnueabihf/dist/include/js/Value.h:1362
#3 mozJSComponentLoader::Import (this=this@entry=0x67a08000, registryLocation=..., targetValArg=..., targetValArg@entry=..., cx=cx@entry=0x76a56000,
    optionalArgc=optionalArgc@entry=0 '\000', retval=retval@entry=...) at /home/jdonald/firefox-59.0.1+build1/js/xpconnect/loader/mozJSComponentLoader.cpp:980
#4 0x717e8c64 in nsXPCComponents_Utils::Import (this=<optimized out>, registryLocation=..., targetObj=..., cx=0x76a56000, optionalArgc=0 '\000', retval=...)
    at /home/jdonald/firefox-59.0.1+build1/js/xpconnect/src/XPCComponents.cpp:2297
#5 0x7119c544 in NS_InvokeByIndex (that=0x1, methodIndex=0, paramCount=1742086496, params=0x7effb9f8)
    at /home/jdonald/firefox-59.0.1+build1/xpcom/reflect/xptcall/md/unix/xptcinvoke_arm.cpp:413
#6 0x0000001e in ?? ()

(and ran disas /r 0x717c4bc8,... to confirm that this symbol-laden debug session was still hitting strd r2, r3, [r1])

The most suspicious code is actually the XPCOM invocation. The armhf-specific assembly in https://hg.mozilla.org/mozilla-central/raw-file/tip/xpcom/reflect/xptcall/md/unix/xptcinvoke_arm.cpp is worrisome. It has no way of knowing whether registers d16 through d31 are available, so may allocate the wrong amount of stack space and clobber extra registers when mixing -mfpu=vfpv3-d16 with -mfpu=neon. The code in xptcinvoke_aarch64.cpp is completely different and looks more robust.

Thus, if anyone is trying to set up a build environment on Ubuntu 14.04 Trusty, keep in mind you might still hit the strd r2, r3, [r1] SIGSEGV. On Bionic: the 59.0.1 18.04 build fr...

[herrtimson (herrtimson)]

The fix from 59.0.1 to 59.0.2 is literally two lines, it shouldn't be a problem.

Thank you for all of your effort, what you write about the messy assembler code seems reasonable. The more I read through the comments in that file, the more I wonder how this could ever worked out in the past.

There is a github read-only mirror of firefox code, it has history function to track down change of files and whole folders much easier, in case you need more evidence: https://github.com/mozilla/gecko-dev/tree/master/xpcom/reflect/xptcall/md/unix/

I'm now going to install glibc debug symbols, to see if the debugger is broken.

[James Donald (jdonald)]

Here is a working Firefox 59 deb package cross-compiled for 18.04 armhf: https://www.dropbox.com/s/17ypog6btdq4wj7/firefox_59.0.1%2Bbuild1-0ubuntu1_armhf.deb?dl=0

I have installed and tested on RaspEX (Bionic armhf). I can also confirm it works sandboxed on Ubuntu MATE Xenial and Raspbian Stretch. One way to run on older systems is to unpack locally using ar + tar, download libc6 and libstdc++6 bionic armhf debfiles and untar them as well, then do:

  cd path/to/usr/lib/firefox
  LD_PRELOAD=path/to/usr/lib/arm-linux-gnueabihf/libstdc++.so.6:path/to/lib/arm-linux-gnueabihf/libm.so.6 \
    ./firefox

Cross-compiling via dpkg-buildpackage is fraught of issues, and similar to what Chituc experienced the most difficult part was getting rustc to properly cross-compile in all cases. I posted my main workaround for a host/target header bug here: https://github.com/rust-lang-nursery/rust-bindgen/issues/1229

To prevent the Skia misaligned instruction crash, I ended up editing gfx/skia/generate_mozbuild.py to force SK_JUMPER_USE_ASSEMBLY to False. However, there's stubborn code at the top of SkJumper.cpp that ends up turning it on again so I had to override that too. Telling Skia to use its non-asm fallback mode should be better than disabling Skia altogether as we had for Firefox 57.

I tested some of our earlier theories too, and so far building Skia with clang, everything with -thumb, -fstack-protector-strong, --enable-rust-simd or other suggestions above do not seem to help at all with the Skia SIGILL nor the strd r2, r3, [r1] SIGSEGV.

As for that strd r2, r3, [r1] crash, not much progress there aside from our theory that it's in the XPCOM armhf-specific code. I'm merely avoiding it by building in a 18.04 Docker container, but still no idea why that works.

[Chituc Georgian (dianaxxyyzz)]

Great , thank you ! I have a special build of skia dir I will try this into your build and see if it will work with skia asm anabled . Thanks again for hard work !

[herrtimson (herrtimson)]

This patch should make it work! I was more lucky than anything else, because there is a compile error with firefox-60.0 betas on armhf which made me find https://bugzilla.mozilla.org/show_bug.cgi?id=1434526

[herrtimson (herrtimson)]

toolchain used is:

glibc-2.25, binutils-2.29.1, gcc-7.3.0
rust-1.24.1, cargo 0.25.0, and llvm/clang-5

please test this with trusty toolchain!

[James Donald (jdonald)]

@herrtimson are you expecting revert-128661.patch to fix the strd r2, r3, [r1] startup segfault on xenial/trusty, the Skia assembly illegal instruction, or both?

The discussion in bug 1434526 indicates this patch is necessary on Firefox 60 to resolve an armhf build failure on the latest trunk, but that regression appears to be newer than both of the crash symptoms at hand here.

[herrtimson (herrtimson)]

The patch unbreaks the compile of firefox-60 betas on armhf, but at the same time it fixes the startup segfault which I had with the toolchain pasted in #120 with stable firefox.

The skia thing could be another story, therefore it would be helpfull to test build this with a trusty toolchain, or xenial if this is affected as well.

My device where I have trusty installed on is a AC100, it has only 512mb ram, but even worse the sd card slot is broken. I can't attach any fast external usb drive for swap or additional temp. space. The only thing I could do is to copy the whole filesystem over and use my rpi2 instead of, to build a deb package. It is just that I don't have much compile time left to spent on this at the moment, plus I'll first have to learn how to do this the debianish way. So if there is anyone who can build a deb and is willing to test it, please go for it.

[James Donald (jdonald)]

@herrtimson from my tests that patch does not resolve the strd r2, r3, [r1] crash on Xenial. However, I do believe it's entirely possible that it indirectly fixed the issue with your gcc-7 toolchain. I'm starting to wonder if this crash does not have a single cause but rather is a codepath that Firefox goes down if there is one of many possible failures in startup.

In addition to the Bionic build above, this weekend I got a Raspbian Stretch build working: https://www.dropbox.com/s/sdqwuda5eaw0cks/firefox_59.0.2%2Bbuild1-stretch-rpi2_armhf.deb?dl=0 with some overview on the forums: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=150438

I used Bionic sources (deb-src http://ports.ubuntu.com/ubuntu-ports/ bionic main restricted) in a Stretch container (gcc-6). Initially it crashed on the same strd, so I tried switching to clang and then encountered the same "undefined reference to `free'" errors you guys had mentioned. These come about mainly due it not being supported to call free() before declaration in C99, so then the fix was to add #include <stdlib.h> in a couple places.

I don't think I'll attempt another Xenial or Trusty build anytime soon. I'd like to get some fixes documented and patches submitted upstream so that others can start building Firefox armhf across more platforms. Anyone can still run the 18.04 binary I linked earlier on 16.04 (and possibly 14.04) using the LD_PRELOAD trick.

As for how Arch Linux has been working, I've confirmed it's due to --disable-stylo (mentioned a while ago on the Raspberry Pi forum thread). If I add that flag (while leaving SK_JUMPER_USE_ASSEMBLY intact) then Firefox runs. For some reason the Stylo flag does not get listed in about:buildconfig but it's a very different build path when MOZ_STYLO is disabled, and Stylo is still disabled for Android nightly too.

[Chris Worsley (c.worsley)]

@jdonald Wow (& sound a hats being taken off).
That really moves the boat forward - well done!
I did wonder what it was that allowed arch to work, and had a hunch that it would be hiding in plain sight in their PKGBUILD file, if only I could understand it.
Thank you for all that work - & your upstream fix intentions!

[James Donald (jdonald)]

@herrtimson

Here's a 14.04 Trusty build: https://www.dropbox.com/s/977vzb3mu8bf34z/firefox_59.0.2%2Bbuild1-0ubuntu0.14.04.4_armhf.deb?dl=0

I used the same approach with clang + gcc headers inside a Trusty (gcc-4.8) container. Initially I got errors about ::gets() and Stack Overflow sent me down some rabbit holes of attempting to use gcc-4.9 headers or libc++ armhf. Ultimately the workaround was to comment out the offending line in gcc headers, and also fix a constexpr bug in the <complex> header. These kinds of hacks are something one can be more comfortable doing inside a Docker container.

I don't have a 14.04 armhf setup but I did test this on Raspbian Stretch. Would be great if you could try it out on your system. Note initially it used up all my Pi's RAM so I had to add 1 GB of swap.

For others, here is a 16.04 Xenial build: https://www.dropbox.com/s/ogdut56m2i0k0t9/firefox_59.0.2%2Bbuild1-0ubuntu0.16.04.3_armhf.deb?dl=0

[herrtimson (herrtimson)]

Ah, cool thing! Could you please post a split patch against the debian file, so that I can see what you have changed exactly and how?

[herrtimson (herrtimson)]

With debian file I mean the xz file which has all the patches, build scripts and so on, it should be called firefox_59.0.2+build1-0ubuntu0.14.04.4.debian.tar.xz

[Lars Nyqvist (ulos)]

Fine!!!!!!
I installed this to my new tinkerboard ubuntu 16.04, Armbian kernel 4.4.120.
Seems to work fine.

Best regards Lars Nyqvist

On 5/4/18, James Donald <email address hidden> wrote:
> @herrtimson
>
> Here's a 14.04 Trusty build:
> https://www.dropbox.com/s/977vzb3mu8bf34z/firefox_59.0.2%2Bbuild1-0ubuntu0.14.04.4_armhf.deb?dl=0
>
> I used the same approach with clang + gcc headers inside a Trusty
> (gcc-4.8) container. Initially I got errors about ::gets() and Stack
> Overflow sent me down some rabbit holes of attempting to use gcc-4.9
> headers or libc++ armhf. Ultimately the workaround was to comment out
> the offending line in gcc headers, and also fix a constexpr bug in the
> <complex> header. These kinds of hacks are something one can be more
> comfortable doing inside a Docker container.
>
> I don't have a 14.04 armhf setup but I did test this on Raspbian
> Stretch. Would be great if you could try it out on your system. Note
> initially it used up all my Pi's RAM so I had to add 1 GB of swap.
>
> For others, here is a 16.04 Xenial build:
> https://www.dropbox.com/s/ogdut56m2i0k0t9/firefox_59.0.2%2Bbuild1-0ubuntu0.16.04.3_armhf.deb?dl=0
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1733340).
> https://bugs.launchpad.net/bugs/1711337
>
> Title:
> Firefox crashes at start on armv7L after 55.0.1 update
>
> Status in Mozilla Firefox:
> Expired
> Status in firefox package in Ubuntu:
> Confirmed
>
> Bug description:
> Firefox always crashes when launched after the 55.0.1 update on an
> Orange Pi PC Plus (a single-board computer similar to a Raspberry Pi),
> even in safe mode.
>
> I did a fresh install of Armbian (a Ubuntu Xenial 16.04 re-spin for
> ARM single-board computer) on a similar board (Orange Pi Plus 2e),
> installed Firefox and experienced the same problem--it won't load
> without crashing.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 16.04
> Package: firefox 55.0.1+build2-0ubuntu0.16.04.2
> Uname: Linux 3.4.113-sun8i armv7l
> AddonCompatCheckDisabled: False
> AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.25.
> ApportVersion: 2.20.1-0ubuntu2.10
> Architecture: armhf
> AudioDevicesInUse:
> USER PID ACCESS COMMAND
> /dev/snd/controlC1: jim 1138 F.... pulseaudio
> /dev/snd/controlC0: jim 1138 F.... pulseaudio
> BuildID: 20170814194718
> Card0.Amixer.info:
> Card hw:0 'audiocodec'/'audiocodec'
> Mixer name : ''
> Components : ''
> Controls : 12
> Simple ctrls : 12
> Card1.Amixer.info:
> Card hw:1 'sndhdmi'/'sndhdmi'
> Mixer name : ''
> Components : ''
> Controls : 1
> Simple ctrls : 1
> Card1.Amixer.values:
> Simple mixer control 'hdmi audio format Function',0
> Capabilities: enum
> Items: 'null' 'pcm' 'AC3' 'MPEG1' 'MP3' 'MPEG2' 'AAC' 'DTS' 'ATRAC'
> 'ONE_BIT_AUDIO' 'DOLBY_DIGITAL_PLUS' 'DTS_HD' 'MAT' 'WMAPRO'
> Item0: 'pcm'
> Channel: Unavailable
> CurrentDesktop: XFCE
> Date: Thu Aug 17 05:37:00 2017
> Extensions: extensions.sqlite corrupt or missing
> ForcedLayersAccel: False
> Incompa...

[James Donald (jdonald)]

@herrtimson I looked in the tarball and don't even see mention of my use of clang in mozconfig.

Note I'm using a procedure similar to what Chituc outlined. If I kick off dpkg-buildpackage, interrupt it, then edit the generated backend.mk I would not expect that to appear in patch files. It's also possible that due to use of the -b flag it skips version control comparisons, so the C++ source diffs are not appearing either.

For now I have documented pieces manually in this repo: https://github.com/jdonald/firefox-armhf

I'll update that repo as more steps come to mind. firefox_59.0.2.build1-0ubuntu0.14.04.4.debian.tar.xz is available for download there in the releases tab if you'd like to take a look through it.

[Lars Nyqvist (ulos)]

Hi Donald,
I am writing this e-mail using your new
firefox_59.0.2+build1-0ubuntu0.14.04.4_armhf.deb version on a
TinkerOS, Debian GNU/Linux 9.4 (stretch) , kernel 4.4.71+
I started firefox in Downloads folder from terminal and got the following lines:
linaro@tinkerboard:~/Downloads$ firefox
[fresh] [error] probe_ppp_module, can't find libppGoogleNaClPluginChrome.so
[fresh] [error] probe_ppp_module, can't find libppGoogleNaClPluginChrome.so
[fresh] [error] probe_ppp_module, can't find libppGoogleNaClPluginChrome.so
[fresh] [error] probe_ppp_module, can't find libppGoogleNaClPluginChrome.so
[fresh] [error] probe_ppp_module, can't find libppGoogleNaClPluginChrome.so
[fresh] [error] probe_ppp_module, can't find libppGoogleNaClPluginChrome.so

Best regards and than you again Lars Nyqvist

On 5/5/18, James Donald <email address hidden> wrote:
> @herrtimson I looked in the tarball and don't even see mention of my use
> of clang in mozconfig.
>
> Note I'm using a procedure similar to what Chituc outlined. If I kick
> off dpkg-buildpackage, interrupt it, then edit the generated backend.mk
> I would not expect that to appear in patch files. It's also possible
> that due to use of the -b flag it skips version control comparisons, so
> the C++ source diffs are not appearing either.
>
> For now I have documented pieces manually in this repo:
> https://github.com/jdonald/firefox-armhf
>
> I'll update that repo as more steps come to mind.
> firefox_59.0.2.build1-0ubuntu0.14.04.4.debian.tar.xz is available for
> download there in the releases tab if you'd like to take a look through
> it.
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1733340).
> https://bugs.launchpad.net/bugs/1711337
>
> Title:
> Firefox crashes at start on armv7L after 55.0.1 update
>
> Status in Mozilla Firefox:
> Expired
> Status in firefox package in Ubuntu:
> Confirmed
>
> Bug description:
> Firefox always crashes when launched after the 55.0.1 update on an
> Orange Pi PC Plus (a single-board computer similar to a Raspberry Pi),
> even in safe mode.
>
> I did a fresh install of Armbian (a Ubuntu Xenial 16.04 re-spin for
> ARM single-board computer) on a similar board (Orange Pi Plus 2e),
> installed Firefox and experienced the same problem--it won't load
> without crashing.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 16.04
> Package: firefox 55.0.1+build2-0ubuntu0.16.04.2
> Uname: Linux 3.4.113-sun8i armv7l
> AddonCompatCheckDisabled: False
> AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.25.
> ApportVersion: 2.20.1-0ubuntu2.10
> Architecture: armhf
> AudioDevicesInUse:
> USER PID ACCESS COMMAND
> /dev/snd/controlC1: jim 1138 F.... pulseaudio
> /dev/snd/controlC0: jim 1138 F.... pulseaudio
> BuildID: 20170814194718
> Card0.Amixer.info:
> Card hw:0 'audiocodec'/'audiocodec'
> Mixer name : ''
> Components : ''
> Controls : 12
> Simple ctrls : 12
> Card1.Amixer.info:
> Card hw:1 'sndhdmi'/'sndhdmi'
> Mixer name : ''
> Components : ''
> Controls : 1
> Simple ctrls : 1
> Card1.A...

[herrtimson (herrtimson)]

Well, for me it doesn't really work. Downloaded this deb file from the github repo: firefox_59.0.2.build1-0ubuntu0.14.04.4_armhf.deb , installed it with dpkg -i , and got the error of 'ungültiger maschienenbefehl', which I am not sure what it does mean at all. Could be illegal assembler instruction.

Oh, and how do I revert this to the ubuntu one? *_*

[James Donald (jdonald)]

Lars, I went to sites like Gmail or Youtube that I thought had a chance of using optional NaCl, but I haven't been able to reproduce those error messages. I don't think Firefox supports NaCl, even with a plugin. Is it possible you imported browser settings from Chromium and it happened to include their NaCl extension? You could rule this out by moving or deleting ~/.mozilla/firefox then trying again.

herrtimson, an illegal instruction would indicate your system lacks NEON or ARMv7 altogether, the same reason Firefox does not run on ARMv6 boards like Raspberry Pi B+ or Pi Zero. To check, cat /proc/cpuinfo

There's also a chance that ungültiger maschienenbefehl means bad interpreter or not in executable format / File format not recognized. You can troubleshoot this with the ldd and file commands. Compare properties of /usr/lib/firefox/firefox to any working command under /usr/bin

> Oh, and how do I revert this to the ubuntu one? *_*

sudo apt-get purge firefox
sudo apt-get install firefox

[Lars Nyqvist (ulos)]

James,
Sorry, I had to report that the bug appeared only when starting
Firefox the first time since then everything is ok.

Regards Lars

On 5/6/18, James Donald <email address hidden> wrote:
> Lars, I went to sites like Gmail or Youtube that I thought had a chance
> of using optional NaCl, but I haven't been able to reproduce those error
> messages. I don't think Firefox supports NaCl, even with a plugin. Is it
> possible you imported browser settings from Chromium and it happened to
> include their NaCl extension? You could rule this out by moving or
> deleting ~/.mozilla/firefox then trying again.
>
> herrtimson, an illegal instruction would indicate your system lacks NEON
> or ARMv7 altogether, the same reason Firefox does not run on ARMv6
> boards like Raspberry Pi B+ or Pi Zero. To check, cat /proc/cpuinfo
>
> There's also a chance that ungültiger maschienenbefehl means bad
> interpreter or not in executable format / File format not recognized.
> You can troubleshoot this with the ldd and file commands. Compare
> properties of /usr/lib/firefox/firefox to any working command under
> /usr/bin
>
>> Oh, and how do I revert this to the ubuntu one? *_*
>
> sudo apt-get purge firefox
> sudo apt-get install firefox
>
> --
> You received this bug notification because you are subscribed to a
> duplicate bug report (1733340).
> https://bugs.launchpad.net/bugs/1711337
>
> Title:
> Firefox crashes at start on armv7L after 55.0.1 update
>
> Status in Mozilla Firefox:
> Expired
> Status in firefox package in Ubuntu:
> Confirmed
>
> Bug description:
> Firefox always crashes when launched after the 55.0.1 update on an
> Orange Pi PC Plus (a single-board computer similar to a Raspberry Pi),
> even in safe mode.
>
> I did a fresh install of Armbian (a Ubuntu Xenial 16.04 re-spin for
> ARM single-board computer) on a similar board (Orange Pi Plus 2e),
> installed Firefox and experienced the same problem--it won't load
> without crashing.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 16.04
> Package: firefox 55.0.1+build2-0ubuntu0.16.04.2
> Uname: Linux 3.4.113-sun8i armv7l
> AddonCompatCheckDisabled: False
> AlsaVersion: Advanced Linux Sound Architecture Driver Version 1.0.25.
> ApportVersion: 2.20.1-0ubuntu2.10
> Architecture: armhf
> AudioDevicesInUse:
> USER PID ACCESS COMMAND
> /dev/snd/controlC1: jim 1138 F.... pulseaudio
> /dev/snd/controlC0: jim 1138 F.... pulseaudio
> BuildID: 20170814194718
> Card0.Amixer.info:
> Card hw:0 'audiocodec'/'audiocodec'
> Mixer name : ''
> Components : ''
> Controls : 12
> Simple ctrls : 12
> Card1.Amixer.info:
> Card hw:1 'sndhdmi'/'sndhdmi'
> Mixer name : ''
> Components : ''
> Controls : 1
> Simple ctrls : 1
> Card1.Amixer.values:
> Simple mixer control 'hdmi audio format Function',0
> Capabilities: enum
> Items: 'null' 'pcm' 'AC3' 'MPEG1' 'MP3' 'MPEG2' 'AAC' 'DTS' 'ATRAC'
> 'ONE_BIT_AUDIO' 'DOLBY_DIGITAL_PLUS' 'DTS_HD' 'MAT' 'WMAPRO'
> Item0: 'pcm'
> Channel: Unavailable
> CurrentDesktop: XFCE
> Date: Thu Aug 17 05:37:00 2017
> Extensions: extension...

[Lars Nyqvist (ulos)]

Hi James, I have tested your
firefox_59.0.2+build1-0ubuntu0.14.04.4_armhf.deb on a
Debian GNU/Linux 9.4 (stretch) and Rockmyy's
Linux tinkerboard 4.17.0-rc3-RockMyy-Seventeen

Any push to the Sound slider quiets sound permanently but the mute
button works ok when
looking a video from youtube.com, areena.yle.fi

Chromium (Version 63.0.3239.84 (Developer Build) built on Debian 9.3,
running on Debian 9.4 (32-bit)) has no problems with the slider.

I know that 4.17.0-rc3 is not supported yet but a problem may lay on the path.

Regards Lars

[Gabriele Svelto (gsvelto)]

Hi everybody, Mozilla developer here, I've come across this after we noticed the crash spike on our servers. The relevant bug is here: https://bugzilla.mozilla.org/show_bug.cgi?id=1452128

Unfortunately the symbols for this package weren't uploaded to our servers so we don't have a useful stack trace to work on. If Firefox' packager could get in touch with us we could debug this together.

[sam tygier (samtygier)]

firefox 60.0+build2-0ubuntu1 is launching for me without crashing (on ubuntu-mate 18.04 on rasberrypi 2B). I have not made any other changes.

[Adam Smith (adamsmith)]

Firefox working too on armhf xubuntu 18.04! (Pi 3B)

[kamal (kamallagh)]

Thanks Mathias and Donald (comments 39 & 40)
after update to v61, Firefox crashes when lunched
Now, It works for me after installing v 57.0.4
Thanks a lot again for your help

[herrtimson (herrtimson)]

still fails over and over again with 14.04 and 16.04!

[herrtimson (herrtimson)]

so the next update for thunderbird will carry this error?

[James Donald (jdonald)]

From what I can tell the maintainers have no intention of spending time supporting Ubuntu 16.04 or older. Even when Xenial was the current LTS they weren't responding here, and now that 18.04 has been out for 5 month there's even less motivation to handle older distros.

Having built Firefox 59 from source I think I get some of the reasons why they'd consider this a heavy support task. It's hard to get gcc and Rust to play nice on old cross-compiling toolchains, not to mention that sketchy assembly code for XPCOM. From a practical perspective, user workarounds can go in this order:

1) Use the latest firefox:armhf on 18.04 Bionic, if that's what you're running or you're open to upgrading.

2) Earlier distros like 16.04 or Debian Stretch: Install the latest firefox:arm64 if your system supports AArch64.

3) If armhf is the only option and you're on 16.04, you can still set up the firefox 18.04 deb package from Launchpad. Download its dependencies like libstdc++ and following my instructions in #117 to place the appropriate runtime-linked libraries.

Relying on armhf builds from Bionic in the worst case 3): this avoids having someone to maintain the whole build flow for Xenial and Trusty.

I'm only speaking for Firefox here, but if it all comes down to crashes in libxul.so then maybe much of this is sufficient for Thunderbird too.