QML segfault on arm64 due to builder kernel change

[Timo Jyrinki]

Builders were changed from wily to xenial (4.4) kernel for arm64, and likely due to this the following kind of errors started happening:

https://launchpadlibrarian.net/288487533/buildlog_ubuntu-yakkety-arm64.ubuntu-push-qml_0.1+15.10.20150826.1-0ubuntu2_BUILDING.txt.gz

They happen when executing QML code.

I've ruled out that reverting to previous qtbase and qtdeclarative versions don't affect this, so it's not coming from a Qt change. I've also verified there's no problem eg on our Bq tablet running the same tests on xenial+overlay. There are currently no found changes at least in Qt upstream to backport arm64 related fixes.

Cause is unknown. But the same problem is there on vivid, xenial and yakkety builds:
https://launchpadlibrarian.net/288516465/buildlog_ubuntu-yakkety-arm64.online-accounts-api_0.1+16.10.20161006.1-0ubuntu1_BUILDING.txt.gz
https://launchpadlibrarian.net/288515705/buildlog_ubuntu-xenial-arm64.online-accounts-api_0.1+16.04.20161006.1-0ubuntu1_BUILDING.txt.gz
https://launchpadlibrarian.net/288515510/buildlog_ubuntu-vivid-arm64.online-accounts-api_0.1+15.04.20161006.1-0ubuntu1_BUILDING.txt.gz

This means it does not matter if it's Qt 5.4 or 5.6, GCC 4.9, 5 or 6, or glibc 2.21, 2.23 or 2.24.

More logs from affected packages:
https://launchpadlibrarian.net/288493058/buildlog_ubuntu-yakkety-arm64.qtdeclarative-opensource-src_5.6.1-7ubuntu3~1_BUILDING.txt.gz
https://launchpadlibrarian.net/288347391/buildlog_ubuntu-xenial-arm64.webbrowser-app_0.23+16.04.20161005.1-0ubuntu1_BUILDING.txt.gz
https://launchpadlibrarian.net/288385063/buildlog_ubuntu-vivid-arm64.qmenumodel_0.2.10+15.04.20161005.1-0ubuntu1_BUILDING.txt.gz

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1630906

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Michael Zanetti (mzanetti)]

This happens in launchpad builders. I don't think it is possible for us to log in and run apport-collect. Also there are many logs linked in the bug description. Putting back to Confirmed.

[Timo Jyrinki (timo-jyrinki)]

I got a temporary access (now revoked) to an arm64 box where I could reproduce the segfault by running qmlplugindump.

Here's the backtrace:
QV4::Object::defineReadonlyProperty (this=this@entry=0x0, name=0x450bf8, value=...) at jsruntime/qv4object.cpp:184
184 jsruntime/qv4object.cpp: No such file or directory.
(gdb) bt
#0 QV4::Object::defineReadonlyProperty (this=this@entry=0x0, name=0x450bf8, value=...) at jsruntime/qv4object.cpp:184
#1 0x0000ffffb7a9f774 in QV4::ObjectPrototype::init (this=0x0, v4=v4@entry=0x4508f0, ctor=0x0) at jsruntime/qv4objectproto.cpp:84
#2 0x0000ffffb7a5a20c in QV4::ExecutionEngine::ExecutionEngine (this=0x4508f0, factory=<optimized out>)
    at jsruntime/qv4engine.cpp:367
#3 0x0000ffffb7b86d94 in QV8Engine::QV8Engine (this=0x4502a0, qq=<optimized out>) at qml/v8/qv8engine.cpp:144
#4 0x0000ffffb7a25de8 in QJSEngine::QJSEngine (this=0xfffffffff298, dd=..., parent=<optimized out>) at jsapi/qjsengine.cpp:201
#5 0x0000ffffb7af738c in QQmlEngine::QQmlEngine (this=0xfffffffff298, parent=0x0) at qml/qqmlengine.cpp:927
#6 0x00000000004067b8 in main (argc=0, argv=<optimized out>) at main.cpp:1041

Where the code in question is http://code.qt.io/cgit/qt/qtdeclarative.git/tree/src/qml/jsruntime/qv4object.cpp?h=5.6.1#n184

I hope that helps somehow.

[Timo Jyrinki (timo-jyrinki)]

For completeness, also the other copy-pastes I made:

Thread 1 "qmlplugindump" received signal SIGSEGV, Segmentation fault.
QV4::Object::defineReadonlyProperty (this=this@entry=0x0, name=0x450bf8, value=...) at jsruntime/qv4object.cpp:184
184 jsruntime/qv4object.cpp: No such file or directory.
(gdb) info frame
Stack level 0, frame at 0xffffffffee00:
 pc = 0xffffb7a99994 in QV4::Object::defineReadonlyProperty (jsruntime/qv4object.cpp:184); saved pc = 0xffffb7a9f774
 called by frame at 0xffffffffee90
 source language c++.
 Arglist at 0xffffffffedb0, args: this=this@entry=0x0, name=0x450bf8, value=...
 Locals at 0xffffffffedb0, Previous frame's sp is 0xffffffffee00
 Saved registers:
  x19 at 0xffffffffedc0, x20 at 0xffffffffedc8, x21 at 0xffffffffedd0, x29 at 0xffffffffedb0, x30 at 0xffffffffedb8
(gdb) up
#1 0x0000ffffb7a9f774 in QV4::ObjectPrototype::init (this=0x0, v4=v4@entry=0x4508f0, ctor=0x0) at jsruntime/qv4objectproto.cpp:84
84 jsruntime/qv4objectproto.cpp: No such file or directory.
(gdb) info frame
Stack level 1, frame at 0xffffffffee90:
 pc = 0xffffb7a9f774 in QV4::ObjectPrototype::init (jsruntime/qv4objectproto.cpp:84); saved pc = 0xffffb7a5a20c
 called by frame at 0xffffffffefe0, caller of frame at 0xffffffffee00
 source language c++.
 Arglist at 0xffffffffee00, args: this=0x0, v4=v4@entry=0x4508f0, ctor=0x0
 Locals at 0xffffffffee00, Previous frame's sp is 0xffffffffee90
 Saved registers:
  x19 at 0xffffffffee10, x20 at 0xffffffffee18, x21 at 0xffffffffee20, x22 at 0xffffffffee28, x23 at 0xffffffffee30,
  x24 at 0xffffffffee38, x25 at 0xffffffffee40, x26 at 0xffffffffee48, x27 at 0xffffffffee50, x28 at 0xffffffffee58,
  x29 at 0xffffffffee00, x30 at 0xffffffffee08

[Timo Jyrinki (timo-jyrinki)]

Correction to the code url: http://code.qt.io/cgit/qt/qtdeclarative.git/tree/src/qml/jsruntime/qv4object.cpp?h=5.6.2#n184 (5.6.1 branch doesn't exist over there, but code seems unchanged)

[Timo Jyrinki (timo-jyrinki)]

Albert pointed to https://codereview.qt-project.org/#/c/169892

-> https://bileto.ubuntu.com/#/ticket/2055 (xenial and yakkety currently building)

I need to stop for today but if someone has time please check if 5.4 backport is possible.

[Timo Jyrinki (timo-jyrinki)]

xenial and yakkety needed symbol updates but now they have compiled and work at least on desktop, and the arm64 build finished so it's likely the arm64 build issue would be solved now. I've asked QA to look into it.

[Timo Jyrinki (timo-jyrinki)]

The linux kernel part of this is setting of CONFIG_ARM64_VA_BITS=48. Also reverting of just that in the 4.4 kernel would work.

[Timo Jyrinki (timo-jyrinki)]

Unfortunately the upstream fix seems to, while fixing arm64, cause segfaults in Unity 8 on armhf and i386:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-yakkety-ci-train-ppa-service-2055/yakkety/armhf/u/unity8/20161007_180848@/log.gz
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-yakkety-ci-train-ppa-service-2055/yakkety/i386/u/unity8/20161009_160829@/log.gz

[Albert Astals Cid (aacid)]

> Unfortunately the upstream fix seems to, while fixing arm64, cause segfaults in Unity 8 on armhf and i386:

It seems we need rebuilds of everything that uses qtdeclarative-private internal headers as those changed.

[Timo Jyrinki (timo-jyrinki)]

Otherwise "ok" so far regarding rebuilds but unmodified UITK starts to fail two subtests on arm64: bug #1631941. Those did pass before the builder kernel update, and it's now not sure where those new failures come from. But they do happen on both xenial and yakkety: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2055/+packages

[Timo Jyrinki (timo-jyrinki)]

I've now also asked on #ubuntu-kernel if CONFIG_ARM64_VA_BITS=48 is here to stay for the 4.4 kernel or not, pointing to this bug. It sounds like no-one is currently considering going back to the 4.2 kernel on the arm64 builders at least.

Meanwhile I'm collecting rebuilds of qtdeclarative-abi-5-6-1 users to the 2055 silo, although it won't help yakkety as such as it's being released and the qtdeclarative update may not be SRU material.

[Łukasz Zemczak (sil2100)]

As per the agreement, the xenial-based builders seem to be now running a newer custom kernel (4.4.0-42-generic) that seems to no longer segfault on QML code for arm64.

I suppose we can set the kernel task as Fix Released? Since I suppose CONFIG_ARM64_VA_BITS=48 is the right way to go, with Qt just needing to support it properly. This way at least we won't need to think about SRUing it to xenial to get things unblocked. Also, our touch arm64 xenial devices are unaffected as they're still using the old kernel, so no blockers here.

[Colin Watson (cjwatson)]

The kernel running on the builders is a custom fork at the moment, which isn't a good long-term situation, so please don't mark the kernel task here as Fix Released.

[Timo Jyrinki (timo-jyrinki)]

It sounded to me the old value would be around 41 or so, and the motivation to increase was a need to have it only slightly bigger? Meanwhile, the Qt patch is about freeing "2 more bits" so maybe 46 or 47 would work too, unless 48 is the magical "good" value that is wanted to be had.

[Olivier Tilloy (osomon)]

It looks like I’m being bit by this problem again. Building webbrowser-app packages fails on xenial+overlay arm64. Tests that execute QML code segfault. Attaching a log file.

Note that the same tests all pass when building on zesty arm64.

[Timo Jyrinki (timo-jyrinki)]

This is fixed in zesty in Qt 5.7.1.

It should be eventually part of also LTS Qt 5.6.3 release some time later in first half of 2017.

[Timo Jyrinki (timo-jyrinki)]

(...and meanwhile the workarounded builder kernel not using the new setting is still required)

[Timo Jyrinki (timo-jyrinki)]

This is now fixed on zesty and xenial stable-phone-overlay PPA, but notably not on normal xenial. xenial SRUs could still be affected, although SRUs having affected tests run during build time might be rare.