[arm] building non-PIC libraries fails under ARMv7 mode

[Michael Casadevall]

Binary package hint: gcc-4.4

Under lucid, GCC can no longer build non-PIC shared libraries, attempting to build a non-PIC library causes:

/usr/bin/ld: .libs/hello.o: relocation R_ARM_THM_MOVW_ABS_NC against `a local symbol' can not be used when making a shared object; recompile with -fPIC error

According to the binutils mailing list (http://old.nabble.com/Detect-ARM-MOVW-MOVT-relocations-in-shared-library-links-td23266269.html) this error is tripped as a safeguard when an object thats compiled for PIC attempts to load from load an absolute symbol address. Its not clear from the list archives if building non-PIC shared libraries is still supported as of ARMv7, or if this is a regression in the toolchain. The issue disappears if -march=armv6 is passed into the CFLAGS.

Reproduction Instructions:
I've attached a copy of the GNU hell test which is used as part of the libtool test suite, and can be used to easily reproduce the problem on a lucid/armel system. Building for ARMv7 mode fails with a relocation error from the linker. Passing -march=armv6 in the CFLAGS allows the build to succeed and the resulting libraries and binaries work as expected:

To build for ARMv7
$ ./configure --with-pic=no && make

To build for ARMv6
$ CFLAGS=-march=armv6 ./configure --with-pic=no && make

[Dave Martin (dave-martin-arm)]

It seems that currently the dynamic linker does not support resolving all the relocations which would be required for non-PIC Thumb-2 shared libraries.

Since non-PIC shared libraries are generally pretty inefficient (memory wastage and thrashing in and out of the dynamic linker and kernel as relocations are fixed up CoW, TLB and cache pollution etc., etc.) I think building such a library would usually be considered an error. This is true on all architectures.

Do we have any real-world examples of packages which build non-PIC shared libraries, or an argument for a case where they might be desirable?

[Michael Casadevall (mcasadevall)]

ffmpeg defaults to non-PIC shared libraries on architectures that support it. Some Googling turned up this blog post: http://hardwarebug.org/2009/01/02/shared-library-woes-and-the-price-of-pic/ (its a bit out of date, the linker now throws an error versus a runtime crash, but there still is a performance gain on non-PIC shared libraries).

[Dave Martin (dave-martin-arm)]

Unfortunately, resolving the problem is not as simple as the article makes out, although silently building libraries containing relocations ld.so can't handle is clearly bad and it's a good thing that was fixed.

The article also doesn't comment on the costs of non-PIC shared libraries (see my last post)

IIUC, non-PIC shared libraries were never really deliberately supported for ARM, they just worked by coincidence.

The difference between ARMv7 and v6 and earlier is that ARMv7 will load 32-bit values into registers directly using the MOVW and MOVT instructions, rather than using a literal pool entry. The impact is that the code executes faster and eliminates some D-cache and D-TLB pollution; however, because relocating these references involves relocating instructions instead of a literal pool data word it would place an extra load on the dynamic linker.

Making this work correctly involves:
  * adding ld.so support for all the text relocations which may appear in static binaries in general (which may be more than just R_ARM_MOVW and R_ARM_MOVT);
  * enabling ld.so to do code relocations correctly (with the implied cache flushing requirements etc.) – I believe this is not implemented at present;
  * getting various complexities right in connection with ELF symbol binding types and symbol preemption
  * changing binutils' idea of what relocation types it should allow in shared libraries

This isn't trivial and requires some concerted effort between binutils and eglibc communities. There's also a risk that implementing these features correctly in the dynamic linker could have an impact on the performance of dynamic linking globally, if not done carefully.

The ELF specifications seem to make no direct comment on whether non-PIC shared libraries should work or not, but there's a defacto assumption from non-RISC architectures (where code relocations are simpler) that they do. It would be interesting to understand how this is currently handled for other RISC architectures such as MIPS and SPARC... but it seems not to be a common problem because very few apps build this way.

[Dave Martin (dave-martin-arm)]

Clarification: for R_ARM_MOVW and R_ARM_MOVT, read R_ARM_MOVW_*, R_ARM_MOVT_*, R_ARM_THM_MOVW_* and R_ARM_THM_MOVT_*.

[Loïc Minier (lool)]

FTR, I had to force -fPIC -DPIC in the ffmpeg build I intend to push; the noopt flavor (main flavor) wouldn't build with our new toolchain defaults. This will be uploaded today or tomorrow.

[Loïc Minier (lool)]

NB: there is a requirement in the Debian and Ubuntu policies to build shared libraries with PIC anyway; the only case where non-PIC is allowed is for performance reasons, typically mpeg2dec or ffmpeg on i386.

[Dave Martin (dave-martin-arm)]

Did you have any feel for the performance delta? If it's really only 2% that is not too concerning; but if it's greater than that it might be more interesting...

[Loïc Minier (lool)]

I'm afraid I didn't run it at all; sorry -- was looking at fixing the build

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jean-Luc Aufranc (jean-luc-aufranc)]

I have the same problem compiling berkelium with linaro toolchain (gcc 4.5.2) on Ubuntu natty.

/usr/lib/gcc/arm-linux-gnueabi/4.5.2/../../../../arm-linux-gnueabi/bin/ld: /home/jaufranc/edev/beagleboard/berkelium/build/chromium/src/out/Release/obj.target/ui/gfx/../../gfx/ui/gfx/rect.o: relocation R_ARM_THM_MOVW_ABS_NC against `a local symbol' can not be used when making a shared object; recompile with -fPIC
/home/jaufranc/edev/beagleboard/berkelium/build/chromium/src/out/Release/obj.target/ui/gfx/libgfx.a: could not read symbols: Bad value

[Ramana Radhakrishnan (ramana)]

On 11 October 2011 08:52, Jean-Luc Aufranc
<email address hidden> wrote:
> I have the same problem compiling berkelium with linaro toolchain (gcc
> 4.5.2) on Ubuntu natty.
>
> /usr/lib/gcc/arm-linux-gnueabi/4.5.2/../../../../arm-linux-gnueabi/bin/ld: /home/jaufranc/edev/beagleboard/berkelium/build/chromium/src/out/Release/obj.target/ui/gfx/../../gfx/ui/gfx/rect.o: relocation R_ARM_THM_MOVW_ABS_NC against `a local symbol' can not be used when making a shared object; recompile with -fPIC
> /home/jaufranc/edev/beagleboard/berkelium/build/chromium/src/out/Release/obj.target/ui/gfx/libgfx.a: could not read symbols: Bad value

If you expect your library to be used only by 1 executable and 1
process statically link it, if you expect your library to be used by
more than one process across multiple executables, then create a
proper shared library using -fPIC. If you don't do this you'll end up
spraying around lots of DT_TEXTREL entries and each time the shared
library is used in a particular process it will end up creating a full
copy of the page with only the change for the addresses in movw / movt
and end up with a much higher memory footprint than expected rather
than just creating a new copy of the addresses you want in the Global
offset table and therefore cause more overhead than anything else.

I don't think the tools should change to allow this through. If you
really really want to do this use the GCC option -mword-relocations on
your command line and then hopefully the compiler will get it right.
If you have hand crafted assembler that uses movw / movt for addresses
and symbols change it to using constant pools.

cheers
Ramana

>
> --
> You received this bug notification because you are a member of Linaro
> Toolchain Developers, which is subscribed to Linaro Binutils.
> https://bugs.launchpad.net/bugs/503448
>
> Title:
>  [arm] building non-PIC libraries fails under ARMv7 mode
>
> Status in Linaro Binutils:
>  New
> Status in “gcc-4.4” package in Ubuntu:
>  Confirmed
>
> Bug description:
>  Binary package hint: gcc-4.4
>
>  Under lucid, GCC can no longer build non-PIC shared libraries,
>  attempting to build a non-PIC library causes:
>
>  /usr/bin/ld: .libs/hello.o: relocation R_ARM_THM_MOVW_ABS_NC against
>  `a local symbol' can not be used when making a shared object;
>  recompile with -fPIC error
>
>  According to the binutils mailing list (http://old.nabble.com/Detect-
>  ARM-MOVW-MOVT-relocations-in-shared-library-links-td23266269.html)
>  this error is tripped as a safeguard when an object thats compiled for
>  PIC attempts to load from load an absolute symbol address. Its not
>  clear from the list archives if building non-PIC shared libraries is
>  still supported as of ARMv7, or if this is a regression in the
>  toolchain. The issue disappears if -march=armv6 is passed into the
>  CFLAGS.
>
>  Reproduction Instructions:
>  I've attached a copy of the GNU hell test which is used as part of the libtool test suite, and can be used to easily reproduce the problem on a lucid/armel system. Building for ARMv7 mode fails with a relocation error from the linker. Passing -march=armv6 in the CFLAGS allows the build to succeed a...

[Will Newton (will-newton)]

It appears that the gcc/binutils maintainers for ARM do not believe that this is a supported mode of operation so it's a "won't fix".