Android container fails to start when built with the gcc-i686-linux-android toolchain

[Ricardo Salveti]

Build instructions:
$ sudo apt-get build-dep android
$ # if we're on amd64, we need to pull in certain 32-bit packages explicitly
$ sudo apt-get install libx11-dev:i386 libreadline6-dev:i386 libgl1-mesa-dev:i386 zlib1g-dev:i386
$ # and then work around the fact that libgl1-mesa-dev isn't Multi-Arch: same yet
$ sudo ln -s /usr/lib/x86_64-linux-gnu/mesa/libGL.so.1 /usr/lib/x86_64-linux-gnu/libGL.so
$ repo init -u https://code-review.phablet.ubuntu.com/p/aosp/platform/manifest.git -b phablet-4.4.2_r1
$ repo sync
$ apt-get source android-src-vendor
$ mv android-src-vendor-7 vendor
$ sudo apt-get install gcc-i686-linux-android
$ cd build && git fetch https://code-review.phablet.ubuntu.com/aosp/platform/build refs/changes/15/215/2 && git cherry-pick FETCH_HEAD && cd ..
$ source build/envsetup.sh
$ lunch aosp_x86-userdebug
$ make -j<n>

Then use the attached scripts to create and run the x86 emulator.

When booting, you should see:
$ ./run-emulator.sh
emulator: found Android build root: /home/rsalveti/projects/phablet/android/build/phablet
emulator: found Android build out: out/target/product/generic_x86
...
Begin: Running /scripts/init-bottom ... done.
[ 0.723931] init: ureadahead main process (396) terminated with status 5
[ 2.423474] init: /dev/hw_random not found
[ 2.423574] init: cannot open '/initlogo.rle'
[ 2.441086] init: /dev/hw_random not found
[ 2.457236] init: cannot find '/system/etc/install-recovery.sh', disabling 'flash_recovery'
[ 2.457278] init: cannot find '/system/bin/ubuntuappmanager.disabled', disabling 'ubuntuappmanager'
[ 2.457951] init: property 'sys.powerctl' doesn't exist while expanding '${sys.powerctl}'
[ 2.457986] init: powerctl: cannot expand '${sys.powerctl}'
[ 2.458008] init: property 'sys.sysctl.extra_free_kbytes' doesn't exist while expanding '${sys.sysctl.extra_free_kbytes}'
[ 2.458027] init: cannot expand '${sys.sysctl.extra_free_kbytes}' while writing to '/proc/sys/vm/extra_free_kbytes'
[ 2.458156] init: cannot find '/sbin/adbd', disabling 'adbd'
[ 2.463354] healthd: wakealarm_init: timerfd_create failed
 * Starting AppArmor profiles * AppArmor not available as kernel LSM.
                                                                         [fail]
 * Setting up X socket directories... [ OK ]
[ 7.916836] healthd: wakealarm_init: timerfd_create failed
[ 12.227047] healthd: wakealarm_init: timerfd_create failed
[ 17.024366] healthd: wakealarm_init: timerfd_create failed
[ 22.034375] healthd: wakealarm_init: timerfd_create failed
[ 22.974753] init: critical process 'servicemanager' exited 4 times in 4 minutes; rebooting into recovery mode
[ 23.013282] SysRq : Emergency Remount R/O
[ 23.024043] loop: Write error at byte offset 2281701376, length 4096.
[ 23.024090] Buffer I/O error on device loop0, logical block 557056
[ 23.024119] JBD2: Error -5 detected when updating journal superblock for loop0-8.
[ 23.024143] loop: Write error at byte offset 2281701376, length 4096.
[ 23.024161] JBD2: Error -5 detected when updating journal superblock for loop0-8.
[ 23.024180] loop: Write error at byte offset 2281701376, length 4096.
[ 23.024197] JBD2: Error -5 detected when updating journal superblock for loop0-8.
[ 23.024217] loop: Write error at byte offset 0, length 4096.

The container dies because it can't keep servicemanager running.

If you remove gcc-i686-linux-android and rebuild a new image, it works as expected.

[Ricardo Salveti (rsalveti)]

Just use the attached scripts at the root folder (so it can find the image files), and you should be able to create and run the emulator.

[Ricardo Salveti (rsalveti)]

The issue here is that the linker binary crated with the gcc-i686-linux-android toolchain contains a .plt section, when it shouldn't.

The consequence is that once you execute a binary that depends on any shared library, the linker will fail when it tries to initialize itself, because it can't contain any .plt section (and in our case it fails when looking for the gdb hook rtld_db_dlactivity).

If you compare the linker binary with the one created using the pre-built android toolchain, you can easily see that the elf file doesn't contain the .plt section, not causing any issue.

Pre-build:
root@ubuntu-phablet:/# objdump -x /system/bin/linker
...
Dynamic Section:
  PLTGOT 0x0001ffe4
  REL 0x00000224
  RELSZ 0x00000778
  RELENT 0x00000008
  RELCOUNT 0x000000ef
  SYMTAB 0x00000114
  SYMENT 0x00000010
  STRTAB 0x000001a4
  STRSZ 0x00000047
  HASH 0x000001ec
  INIT_ARRAY 0x0001ff00
  INIT_ARRAYSZ 0x00000004
  SYMBOLIC 0x00000000
  FLAGS 0x0000000a
  FLAGS_1 0x00000001

gcc-i686-linux-android:
root@ubuntu-phablet:/# objdump -x /system/bin/linker
...
Dynamic Section:
  PLTGOT 0x0001ffd4
  PLTRELSZ 0x00000028
  JMPREL 0x0000099c
  PLTREL 0x00000011
  REL 0x00000224
  RELSZ 0x00000778
  RELENT 0x00000008
  RELCOUNT 0x000000ee
  SYMTAB 0x00000114
  SYMENT 0x00000010
  STRTAB 0x000001a4
  STRSZ 0x00000047
  HASH 0x000001ec
  INIT_ARRAY 0x0001fee0
  INIT_ARRAYSZ 0x00000004
  FLAGS 0x00000008
  FLAGS_1 0x00000001

[Ricardo Salveti (rsalveti)]

And where it fails:

bionic/linker/linker.cpp:
1303 static bool soinfo_link_image(soinfo* si) {
...
1360 case DT_JMPREL:
1361 si->plt_rel = (Elf32_Rel*) (base + d->d_un.d_ptr);
...
1543 if (si->plt_rel != NULL) {
1544 DEBUG("[ relocating %s plt ]", si->name );
1545 if (soinfo_relocate(si, si->plt_rel, si->plt_rel_count, needed)) {
1546 return false;
1547 }
1548 }

 848 static int soinfo_relocate(soinfo* si, Elf32_Rel* rel, unsigned count,
 849 soinfo* needed[])
...
 874 s = &symtab[sym];
 875 if (ELF32_ST_BIND(s->st_info) != STB_WEAK) {
 876 DL_ERR("cannot locate symbol \"%s\" referenced by \"%s\"...", sym_name, si->name);
 877 return -1;
 878 }

[Ricardo Salveti (rsalveti)]

When comparing the readelf output:
+000201e8 00000101 R_386_32 00008210 rtld_db_dlactivity
+
+Relocation section '.rel.plt' at offset 0x99c contains 5 entries:
+ Offset Info Type Sym.Value Sym. Name
+0001ffe0 00000107 R_386_JUMP_SLOT 00008210 rtld_db_dlactivity
+0001ffe4 00000307 R_386_JUMP_SLOT 00005ba0 free
+0001ffe8 00000507 R_386_JUMP_SLOT 00005c60 calloc
+0001ffec 00000207 R_386_JUMP_SLOT 00005b40 malloc
+0001fff0 00000407 R_386_JUMP_SLOT 00005c00 realloc

rtld_db_dlactivity is just a hook used by gdb so it can be notified about linker activity.

And how it is defined in linker.cpp:
174 extern "C" void __attribute__((noinline)) __attribute__((visibility("default"))) rtld_db_dlactivity();

[Dimitri John Ledkov (xnox)]

Comparing with mako, it has .rel.plt section:

Relocation section '.rel.plt' at offset 0xa08 contains 3 entries:
 Offset Info Type Sym.Value Sym. Name
0000fff4 00000816 R_ARM_JUMP_SLOT 00000000 __gnu_Unwind_Find_exid
0000fff8 00000916 R_ARM_JUMP_SLOT 00000000 __cxa_begin_cleanup
0000fffc 00000a16 R_ARM_JUMP_SLOT 00000000 __cxa_type_match

[Ricardo Salveti (rsalveti)]

In the ARM case I believe the symbols would fit STB_WEAK, otherwise it'd also fail in the same way.

Just confirmed that the results are generated in the final stage of the linking process, and it doesn't depend on the object files used.

Using prebuilts/gcc/linux-x86/x86/i686-linux-android-4.7/bin/i686-linux-android-g++ with the object files created by our toolchain results in a valid linker file.

Using /usr/bin/i686-linux-android-g++ with the object files created by the pre-built toolchain generates a linker with plt.

[Ricardo Salveti (rsalveti)]

The difference is that the -Bsymbolic flag is given to the linker when using the pre-build toolchain (it seems for every shared library).

Our toolchain doesn't use that flag, causing the plt to be generated.

I'm able to successfully build a working linker by forcing this specific flag in Android.mk:
LOCAL_LDFLAGS := -shared -Wl,--exclude-libs,ALL -Wl,-Bsymbolic

But then you get tons of warnings when starting the container, because the same flag wasn't used for the rest of the shared libraries:
WARNING: linker: libdsyscalls.so has text relocations. This is wasting memory and is a security risk. Please fix.
WARNING: linker: libc.so has text relocations. This is wasting memory and is a security risk. Please fix.
WARNING: linker: libm.so has text relocations. This is wasting memory and is a security risk. Please fix.
WARNING: linker: liblog.so has text relocations. This is wasting memory and is a security risk. Please fix.
WARNING: linker: libcutils.so has text relocations. This is wasting memory and is a security risk. Please fix.
WARNING: linker: libusbhost.so has text relocations. This is wasting memory and is a security risk. Please fix.
WARNING: linker: libselinux.so has text relocations. This is wasting memory and is a security risk. Please fix.
WARNING: linker: ls has text relocations. This is wasting memory and is a security risk. Please fix.

[Ricardo Salveti (rsalveti)]

Actually the other warnings listed above is because of DT_TEXTREL. Not sure yet why we get them with our compiler yet.

[Ricardo Salveti (rsalveti)]

So yeah, -Bsymbolic is the reason :-)

Check http://gcc.gnu.org/ml/gcc/2013-03/msg00304.html. And http://gcc.gnu.org/ml/gcc/2013-04/msg00016.html as follow up.

So the prebuilt Android toolchain forces -Bsymbolic for every shared library.

As this is not the case with our toolchain, the linker ends up being built without such flag, causing the original issue.

[Dimitri John Ledkov (xnox)]

\o/

Dynamic Section:
  PLTGOT 0x0001ffe4
  REL 0x00000224
  RELSZ 0x00000778
  RELENT 0x00000008
  RELCOUNT 0x000000ef
  SYMTAB 0x00000114
  SYMENT 0x00000010
  STRTAB 0x000001a4
  STRSZ 0x00000047
  HASH 0x000001ec
  INIT_ARRAY 0x0001ff00
  INIT_ARRAYSZ 0x00000004
  SYMBOLIC 0x00000000
  FLAGS 0x0000000a
  FLAGS_1 0x00000001

[Launchpad Janitor (janitor)]

This bug was fixed in the package android - 20140501-2310-0ubuntu1

---------------
android (20140501-2310-0ubuntu1) utopic; urgency=medium

  * New upstream release
  * Dropping patches already available in upstream:
    - audio_hw.c-make-sure-we-don-t-double-free-if-it-s-th.patch
    - disable-x86-stack-protector.patch
  * debian/patches/0001-core-support-make-4.00.patch:
    - Make it build with make >= 4.0
  * debian/patches/0001-TARGET_linux-x86.mk-forcing-Wl-Bsymbolic.patch:
    - Force -Wl,-Bsymbolic as a workaround for bug LP: #1305315
 -- Ricardo Salveti de Araujo <email address hidden> Mon, 05 May 2014 19:52:36 -0300