cross-linker behaviour differs from native linked

Bug #923779 reported by Riku Voipio on 2012-01-30
This bug affects 2 people
Affects Status Importance Assigned to Milestone
armel-cross-toolchain-base (Ubuntu)
Marcin Juszkiewicz
armhf-cross-toolchain-base (Ubuntu)
binutils (Ubuntu)
binutils-armel-cross (Ubuntu)
binutils-armhf-cross (Ubuntu)

Bug Description

This requires a bit of background to understand. From the man page of ld(1).

           When using ELF or SunOS, one shared library may require another.
           This happens when an "ld -shared" link includes a shared library as
           one of the input files.

           When the linker encounters such a dependency when doing a non-
           shared, non-relocatable link, it will automatically try to locate
           the required shared library and include it in the link, if it is
           not included explicitly. In such a case, the -rpath-link option
           specifies the first set of directories to search. The -rpath-link
           option may specify a sequence of directory names either by
           specifying a list of names separated by colons, or by appearing
           multiple times.

           This option should be used with caution as it overrides the search
           path that may have been hard compiled into a shared library. In
           such a case it is possible to use unintentionally a different
           search path than the runtime linker would do.

           The linker uses the following search paths to locate required
           shared libraries:

           1. Any directories specified by -rpath-link options.

           2. Any directories specified by -rpath options. The difference
               between -rpath and -rpath-link is that directories specified by
               -rpath options are included in the executable and used at
               runtime, whereas the -rpath-link option is only effective at
               link time. Searching -rpath in this way is only supported by
               native linkers and cross linkers which have been configured
               with the --with-sysroot option.

           3. On an ELF system, for native linkers, if the -rpath and
               -rpath-link options were not used, search the contents of the
               environment variable "LD_RUN_PATH".

           4. On SunOS, if the -rpath option was not used, search any
               directories specified using -L options.

           5. For a native linker, the search the contents of the environment
               variable "LD_LIBRARY_PATH".

           6. For a native ELF linker, the directories in "DT_RUNPATH" or
               "DT_RPATH" of a shared library are searched for shared
               libraries needed by it. The "DT_RPATH" entries are ignored if
               "DT_RUNPATH" entries exist.

           7. The default directories, normally /lib and /usr/lib.

           8. For a native linker on an ELF system, if the file
               /etc/ exists, the list of directories found in that

           If the required shared library is not found, the linker will issue
           a warning and continue with the link.

Our concern are the sections 3, 5, 6, 8 and indirectly 7 (/usr/lib/arm-linux-gnueabi isn't apparently considered default directory).

The problem is, that while we search for libraries in /usr/lib/arm-linux-gnueabi, the search is not _recursive_. Take case that we link against -lhi, which in turn uses symbols from -lhi. With a native toolchain, we can just build by giving "gcc -lhi". With current cross-toolchain, that is not enough.

The attached testcase, which succeeds on native ld, will fail with the cross-toolchain with:

/usr/lib/gcc/arm-linux-gnueabi/4.6/../../../../arm-linux-gnueabi/bin/ld: warning:, needed by /usr/lib/arm-linux-gnueabi/, not found (try using -rpath or -rpath-link)
/usr/lib/arm-linux-gnueabi/ undefined reference to `alowcall'
collect2: ld returned 1 exit status

Changing the last line to either:

arm-linux-gnueabi-gcc hiuser.c -llo -lhi -o foouser
arm-linux-gnueabi-gcc hiuser.c -Wl,-rpath-link=/usr/lib/arm-linux-gnueabi -lhi -o foouser

Will make the build succeed.

We have a bunch of options here:

1) patch ld to make cross-ld behave identically to ld. This can be done by setting "NATIVE=yes" in of binutils
2) add /usr/lib/arm-linux-gnueabi to default directories (after quick search didn't find out how). This could still risk breaking if there are packages that use LD_LIBRARY_PATH during buildtime
3) tell people to change their buildscripts to add all libraries to the linking command line. This is tricky, as most people don't select the -lflags them self, they come from pkg-config et all. Further more, people have been actively endorsing doing the complete opposite, eg. removing all unnecessary -lflags to cut down package dependencies.
4) copy the ld wrapper from scratchbox that adds the default directories and LD_LIBRARY_PATH with -rpath-link to the real linkier... I don't think we want that, but just for sake of completeness lets mention it ;)


Riku Voipio (riku-voipio) wrote :
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in armel-cross-toolchain-base (Ubuntu):
status: New → Confirmed
Wookey (wookey) on 2012-04-02
Changed in armel-cross-toolchain-base (Ubuntu):
assignee: nobody → Linaro Developer Platform (linaro-foundations)
Marcin Juszkiewicz (hrw) on 2012-04-03
Changed in armel-cross-toolchain-base (Ubuntu):
assignee: Linaro Developer Platform (linaro-foundations) → Marcin Juszkiewicz (hrw)
status: Confirmed → In Progress
Marcin Juszkiewicz (hrw) wrote :

This version works for me - passed

Wookey (wookey) wrote :

Before this path the default linker script had:
SEARCH_DIR("/usr/x86_64-linux-gnu/lib64"); SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/local/lib64"); S$

with this patch the search dir is:
SEARCH_DIR("/usr/arm-linux-gnueabi/lib"); SEARCH_DIR("/lib/arm-linux-gnueabi"); SEARCH_DIR("/usr/lib/arm-linux-gnueabi");
SEARCH_DIR("/usr/x86_64-linux-gnu/lib64"); SEARCH_DIR("=/usr/local/lib/x86_64-linux-gnu"); SEARCH_DIR("=/usr/local/lib64"); S$

I think the correct search path should be:
 SEARCH_DIR("/lib/arm-linux-gnueabi"); SEARCH_DIR("/usr/lib/arm-linux-gnueabi");SEARCH_DIR("/usr/arm-linux-gnueabi/lib");
(i.e so multiarch dirs are searched before the 'classic cross-build' path.

This solves the immediate problem that native and cross behaviour are different, but I'm concerned that in fact neither of them are doing it right.

If main has symbols in libhi and libhi has symbols in liblo, but main does not have symbols in liblo, then libhi is a build-dep of main but liblo is not. Liblo is a build-dep of libhi. And binutils should be able to link main without liblo being present.

So far as I can see that's not currently happenning. Unless I am misunderstanding something.

Marcin Juszkiewicz (hrw) wrote :

gold is not touched yet

tags: added: patch
Marcin Juszkiewicz (hrw) wrote :

This is better version - previous one broke native build

Wookey (wookey) wrote :

Modified patch to change search dir ordering.

Marcin Juszkiewicz (hrw) wrote :

Fixed version. Allows to build cross and native.

Wookey (wookey) wrote :

Here's an updated version of the test-toolchain script which makes it easy to compare architectures native/cross and generate debug. Run it as test-toolchain <arch>. Not very exciting but more use here than on my drive.

Marcin Juszkiewicz (hrw) wrote :

This patch modifies also gold - not yet checked with native build!

Marcin Juszkiewicz (hrw) wrote :

native build was fine with 4.diff

Marcin Juszkiewicz (hrw) on 2012-04-06
Changed in binutils (Ubuntu):
status: New → Confirmed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package armel-cross-toolchain-base - 1.80

armel-cross-toolchain-base (1.80) precise; urgency=low

  [ Gustavo Alkmim]
  * Updates to make package building under Debian.
  * Copy multiarch headers to standard directory to fix build-eglibc stages.

  [ Marcin Juszkiewicz ]
  * Bumped gcc-4.6 requirement to 4.6.2-16 (due to patches merged).
  * Refreshed Ubuntu and Debian patches.
  * Debian only changes:
    - Automated checking of Linux version (idea from Gustavo).
    - Fixed 'build-linux' step for armhf target
    - clean tmp/ directory after 'build-linux' step to not break other
  * Added patch to binutils to make linker use multiarch directories - LP: #923779
 -- Marcin Juszkiewicz <email address hidden> Fri, 06 Apr 2012 10:38:19 +0200

Changed in armel-cross-toolchain-base (Ubuntu):
status: In Progress → Fix Released
Matthias Klose (doko) wrote :

Riku, please check with the binutils quantal package, what is still needed for this issue

Changed in binutils (Ubuntu):
status: Confirmed → Incomplete

+++ Matthias Klose [2012-09-07 14:33 -0000]:
> Riku, please check with the binutils quantal package, what is still
> needed for this issue

OK. I just tried here as I had quantal chroot and testcode to hand.
Both amd64 and armhf now work the same:

I've not check this in detail, but it looks fine at first glance:

attached are script used for:
sudo ./ amd64 &> quantalamd64link
sudo ./ armhf &> quantalarmhflink
and output files.

Principal hats: Linaro, Emdebian, Wookware, Balloonboard, ARM

Matthias Klose (doko) wrote :

ok, closing

Changed in binutils (Ubuntu):
status: Incomplete → Fix Released
Steve Langasek (vorlon) wrote :

I've continued to see this issue in quantal, so I think this is still open.

Changed in armel-cross-toolchain-base (Ubuntu):
status: Fix Released → Triaged
Matthias Klose (doko) wrote :

from my point of view, the cross hacks are not necessary. the cross binutils just needs to be configured --with-sysroot=/. The only downside is that some remaining "native" libs in /lib and /usr/lib may cause linker warnings, if the ARM libs cannot be found in the the multiarch locations. fixed binutils-armhf-cross so far.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package binutils-armhf-cross - 1.88

binutils-armhf-cross (1.88) raring; urgency=low

  * Remove the add-multiarch-dirs-for-linker patch.
  * Build --with-sysroot=/. LP: #923779.
  * Build-depend on binutils-source (>= 2.23.1-0ubuntu4).
 -- Matthias Klose <email address hidden> Wed, 05 Dec 2012 09:17:21 +0100

Changed in binutils-armhf-cross (Ubuntu):
status: New → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package binutils-armel-cross - 1.88

binutils-armel-cross (1.88) raring; urgency=low

  * Remove the add-multiarch-dirs-for-linker patch.
  * Build --with-sysroot=/. LP: #923779.
  * Build-depend on binutils-source (>= 2.23.1-0ubuntu4).
 -- Matthias Klose <email address hidden> Wed, 05 Dec 2012 09:17:21 +0100

Changed in binutils-armel-cross (Ubuntu):
status: New → Fix Released
Matthias Klose (doko) wrote :

now configured the gcc builds --with-sysroot=/

Changed in armhf-cross-toolchain-base (Ubuntu):
status: New → Fix Released
Changed in armel-cross-toolchain-base (Ubuntu):
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers