libUMP.so from Mali binaries on android breaks down with platform built with -O3

[Kalle Vahlman]

My own builds and build #200 from

  https://android-build.linaro.org/builds/~linaro-android/snowball-ics-gcc46-igloo-stable-blob/

end up not booting correctly due to some kind of linking/missing symbol problem:

D/libEGL ( 2953): loaded /system/lib/egl/libGLES_android.so
E/libEGL ( 2953): load_driver(/system/lib/egl/libEGL_mali.so): Cannot load library: link_image[1936]: 1497 could not load needed library 'libMali.so' for 'libEGL_mali.so' (link_image[1936]: 1497 could not load needed library 'libUMP.so' for 'libMali.so' (reloc_library[1285]: 1497 cannot locate '__aeabi_llsr'...
E/libEGL ( 2953): ))
D/AndroidRuntime( 2953): Shutting down VM
W/dalvikvm( 2953): threadid=1: thread exiting with uncaught exception (group=0xa000f1f8)
E/AndroidRuntime( 2953): FATAL EXCEPTION: main
E/AndroidRuntime( 2953): java.lang.RuntimeException: eglConfig not initialized
E/AndroidRuntime( 2953): at android.view.HardwareRenderer$GlRenderer.initializeEgl(HardwareRenderer.java:571)
E/AndroidRuntime( 2953): at android.view.HardwareRenderer$GlRenderer.initialize(HardwareRenderer.java:504)
E/AndroidRuntime( 2953): at android.view.ViewRootImpl.performTraversals(ViewRootImpl.java:1300)
E/AndroidRuntime( 2953): at android.view.ViewRootImpl.handleMessage(ViewRootImpl.java:2442)
E/AndroidRuntime( 2953): at android.os.Handler.dispatchMessage(Handler.java:99)
E/AndroidRuntime( 2953): at android.os.Looper.loop(Looper.java:137)
E/AndroidRuntime( 2953): at android.app.ActivityThread.main(ActivityThread.java:4424)
E/AndroidRuntime( 2953): at java.lang.reflect.Method.invokeNative(Native Method)
E/AndroidRuntime( 2953): at java.lang.reflect.Method.invoke(Method.java:511)
E/AndroidRuntime( 2953): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:784)
E/AndroidRuntime( 2953): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:551)
E/AndroidRuntime( 2953): at dalvik.system.NativeStart.main(Native Method)
W/ActivityManager( 1632): Force finishing activity com.android.launcher/com.android.launcher2.Launcher

[Thomas Espersson (espersson)]

Is this not seen in build #199 or previous builds? How can you tell what the differences are in build #200 from previous? I tried this link
https://android-build.linaro.org/jenkins/job/linaro-android_snowball-ics-gcc46-igloo-stable-blob/200/changes
But that just says "no changes"

[Kalle Vahlman (kvahlman)]

I'm not sure, I only tested the latest build from there to confirm that it's not just my own build that is broken.

[Thomas Espersson (espersson)]

Kalle, what was the impact when using this build, didn't boot at all or was it booting but with these errors in the log?

The test results from the Linaro build seems to pass (all except CTS):

http://validation.linaro.org/lava-server/dashboard/streams/anonymous/android-daily/bundles/4c6354a4d8f97eda27fbe92e19991b4d781a67b8/36fc4c3c-6d4e-11e1-9729-68b599be548c/

[Kalle Vahlman (kvahlman)]

The symptoms are that UI either does not come up at all, or comes up with a crash notice for launcher.

Which would AFAIK mean that eg. glmark could not be successfully run. However,

  http://validation.linaro.org/lava-server/dashboard/streams/anonymous/android-daily/bundles/4c6354a4d8f97eda27fbe92e19991b4d781a67b8/36fc4c3c-6d4e-11e1-9729-68b599be548c/result/7/

doesn't really reveal what made this test pass. There is "no measurement taken" and no logs nor other info offered.

Further information:

I've now tested builds #200, #195 and #176 and can confirm that #176 is working while the two later ones don't. Looking at the build history at

  https://android-build.linaro.org/jenkins/job/linaro-android_snowball-ics-gcc46-igloo-stable-blob/

something "big" happened after #176 as there are a lot of failed builds after that which could be a hint for the cause of this. I'll try #191 to narrow it down more as well.

[Kalle Vahlman (kvahlman)]

Interestingly #191 and #192 work too, so something happened between builds #192 and #195 to break this.

[Thomas Espersson (espersson)]

Kalle, please update with your findings from the continued investigation

[Kalle Vahlman (kvahlman)]

The problem has been identified and verified to be caused by this change:

  http://review.android.git.linaro.org/#change,1561

In a nutshell, when the image is compiled with -O3 and more recent libgcc than what used to build the mali binaries, some symbols that libUMP.so requries are lost. I've tested that reverting that commit produces a working image.

As for solutions, from IRC discussions:

17:58 < bero> Well, the best thing to do would be to get a recompiled version that uses our compiler and -O3, to make sure it doesn't require __aeabi_llsr to be exported...

but I understood that bero is in the process of finding a way to inject the required implementations in somehow.

[Bernhard Rosenkraenzer (berolinux)]

Yes, currently investigating why -O3 affects this (probably a call to __aeabi_llsr in an underlying library is optimized away by -O3, causing the underlying library not to drag the symbol in at link time (it's supposed to come from libgcc.a)).

There may be a simple fix along the lines of

mv libUMP.so libBrokenUMP.so
arm-linux-androideabi-gcc -shared -o libUMP.so -L. -lBrokenUMP -lgcc

But of course that's rather ugly...

[Patrik Klinger (patrik-klinger)]

Kalle is not working on this bug ... .it should be assigned to someone that is actually working on it. We need to have this fixed ASAP (within 1-2 days).

[Anmar Oueja (anmar)]

libUMP needs to be recompiled by ST-Ericsson. Ideally Patrik Rid should do that. I am assigning it to him.

[Anmar Oueja (anmar)]

Sorry for the misspelled name :-(

[Mathieu Poirier (mathieu.poirier-deactivatedaccount)]

There is a work around to address this problem for 12.03.

[Anmar Oueja (anmar)]

will move to 12.04 release since the work around suffices for the 12.03 release.

[Kalle Vahlman (kvahlman)]

For the record, this seems to have been:

commit fccd4888194b7c7be0dea41eb10a92636175045d
Merge: f5cb4a6 bb3b30c
Author: Mathieu J. Poirier <email address hidden>
Date: Wed Mar 21 16:24:10 2012 +0000

    Merge "snowball: disabling -03 optimzation until we have new mali libs" into linaro_android_4.0.3

in the build.git component.

[Kalle Vahlman (kvahlman)]

I mean the work-around seems to have been...

[Mathieu Poirier (mathieu.poirier-deactivatedaccount)]

The work around is in 'build/core/combo/TARGET_linux-arm.mk' where 'ifneq' have been introduce to use '-03' for all boards except snowball. The additions are currently on line 69 and 86.

[Anmar Oueja (anmar)]

Anmar to check with Graphics WG to see if they can help.

[Anmar Oueja (anmar)]

Anmar pinged the graphics group. They promised they will look at it. However, they are at an important UMM summit now so not expecting an answer for a while.

[Anmar Oueja (anmar)]

I asked Appal to talk to the toolchain group. I assume nothing yet.