dynamic linker does not use DT_RUNPATH for transitive dependencies

[Moritz Hassert]

$ lsb_release -rd
Description: Ubuntu 13.10
Release: 13.10

$ uname -a
Linux mhassert 3.11.0-13-generic #20-Ubuntu SMP Wed Oct 23 07:38:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

$gcc -dumpversion
4.8

$ ld -v
GNU ld (GNU Binutils for Ubuntu) 2.23.52.20130913

$ LC_ALL=C apt-cache policy libc-bin
libc-bin:
  Installed: 2.17-93ubuntu4

* What you expected to happen
Binaries with DT_RPATH or DT_RUNPATH behaving identical in the absence of LD_LIBRARY_PATH

* What happened instead
DT_RUNPATH not searched for transitive dependencies.

When running a binary that depends on custom libraries which in turn depend on custom libraries, hard-coded search paths in DT_RUNPATH behave differently from those in DT_RPATH.
Paths in DT_RPATH are being considered for everything that is dynamically loaded, even dependencies of dependencies. Paths in DT_RUNPATH seem being considered only for direct dependencies of the binary.

Searching the web I think that the one and only difference between DT_RPATH and DT_RUNPATH should be that DT_RPATH is considered _before_ LD_LIBRARY_PATH and DT_RUNPATH _afterwards_. In the absence of LD_LIBRARY_PATH there should be no difference at all.

I stumbled upon this problem when switching from "ld" to "gold" for the linker. The default for ld on Ubuntu 13.10 is "--disable-new-dtags" while the default for gold is "--enable-new-dtags". Therefore ld produces binaries with DT_RPATH and gold ones with DT_RUNPATH.

In the attached minimal example
- the binaries rpath and runpath both depend on libb but not directly on liba.
- libb depends on liba.
- liba and libb are linked without any hard-coded library paths.
- rpath and runpath are linked with hard-coded library paths for both liba and libb
- rpath is linked with --disable-new-dtags (producing DT_RPATH)
- rpath is linked with --enable-new-dtags (producing DT_RUNPATH)

To test, please run make all and observe how "rpath" works while "runpath" fails to find liba at runtime.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Adam Novak (interfect)]

I don't know if this is expected behavior, but it's certainly annoying behavior. I'm writing an app that depends on libtcodxx.so, which in turn depends on libtcod.so. I want to ship them both in the lib directory next to my app. With RUNPATH as "$ORIGIN/lib", it finds the direct dependency (libtcodxx.so) and then doesn't look in the same directory for its dependency. To get it to search there for both, I have to set RPATH to "$ORIGIN/lib", and I suspect that RPATH may not technically be supposed to support the $ORIGIN syntax.

[Matt Whitlock (whitslack)]

This bug (if it's a bug) also affects sys-libs/glibc-2.23-r3 on Gentoo.

The Qt Blog has a post about this issue from 2011:
http://blog.qt.io/blog/2011/10/28/rpath-and-runpath/

This may not be a bug: it is possible that DT_RUNPATH was never intended to be transitive. However, if this is so, then DT_RPATH should not have been deprecated, as DT_RUNPATH does not fully replicate its behavior (at a lower precedence than LD_LIBRARY_PATH).

My use case: I have a cross-compiling toolchain setup to allow me to compile binaries for the Amazon Linux distribution, which has older everything than my host system. To test the compiled binaries on my host system, I set LD_RUN_PATH and -Wl,-dynamic-linker when linking so that the Amazon-versioned libraries are used instead of my host system's libraries. The particular problem I'm experiencing is that libstdc++.so.6 needs libm.so.6, but the latter is not loaded from the proper location when my binary uses DT_RUNPATH instead of DT_RPATH.

Compare:

* When I link my binary with -Wl,--enable-new-dtags (the default on Gentoo):

    $ ldd out/x86_64-amazon-linux-gnu/engine
        linux-vdso.so.1 (0x00007ffeb6bdf000)
        libtcmalloc_minimal.so.4 => /usr/x86_64-amazon-linux-gnu/usr/lib64/libtcmalloc_minimal.so.4 (0x00007f2592d85000)
        libstdc++.so.6 => /usr/lib/gcc/x86_64-amazon-linux-gnu/4.8.3/libstdc++.so.6 (0x00007f2592a53000)
        libgcc_s.so.1 => /usr/lib/gcc/x86_64-amazon-linux-gnu/4.8.3/libgcc_s.so.1 (0x00007f259283c000)
        libpthread.so.0 => /usr/x86_64-amazon-linux-gnu/lib64/libpthread.so.0 (0x00007f259261f000)
        libc.so.6 => /usr/x86_64-amazon-linux-gnu/lib64/libc.so.6 (0x00007f2592274000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f2591f76000)
        /usr/x86_64-amazon-linux-gnu/lib64/ld-linux-x86-64.so.2 (0x00007f2592fcc000)

  Notice that libm.so.6 resolves to the system library in /lib64.

* When I link my binary with -Wl,--disable-new-dtags:

    $ ldd out/x86_64-amazon-linux-gnu/engine
        linux-vdso.so.1 (0x00007ffec67c2000)
        libtcmalloc_minimal.so.4 => /usr/x86_64-amazon-linux-gnu/usr/lib64/libtcmalloc_minimal.so.4 (0x00007fb5208b0000)
        libstdc++.so.6 => /usr/lib/gcc/x86_64-amazon-linux-gnu/4.8.3/libstdc++.so.6 (0x00007fb52057e000)
        libgcc_s.so.1 => /usr/lib/gcc/x86_64-amazon-linux-gnu/4.8.3/libgcc_s.so.1 (0x00007fb520367000)
        libpthread.so.0 => /usr/x86_64-amazon-linux-gnu/lib64/libpthread.so.0 (0x00007fb52014a000)
        libc.so.6 => /usr/x86_64-amazon-linux-gnu/lib64/libc.so.6 (0x00007fb51fd9f000)
        libm.so.6 => /usr/x86_64-amazon-linux-gnu/lib64/libm.so.6 (0x00007fb51faa1000)
        /usr/x86_64-amazon-linux-gnu/lib64/ld-linux-x86-64.so.2 (0x00007fb520af7000)

  Notice that libm.so.6 resolves to the correct library.

[Mario Sánchez Prada (mariospr)]

> I don't know if this is expected behavior, but it's certainly annoying behavior. I'm writing an app that depends on libtcodxx.so, which in turn depends on libtcod.so. I want to ship them both in the lib directory next to my app. With RUNPATH as "$ORIGIN/lib", it finds the direct dependency (libtcodxx.so) and then doesn't look in the same directory for its dependency. To get it to search there for both, I have to set RPATH to "$ORIGIN/lib", and I suspect that RPATH may not technically be supposed to support the $ORIGIN syntax.

It looks like it's the expected behaviour after all. From "Shared Object Dependencies" section in [1]:

"""
The set of directories specified by a given DT_RUNPATH entry is used to find only the immediate dependencies of the executable or shared object containing the DT_RUNPATH entry. That is, it is used only for those dependencies contained in the DT_NEEDED entries of the dynamic structure containing the DT_RUNPATH entry, itself. One object's DT_RUNPATH entry does not affect the search for any other object's dependencies.
"""

Another interesting bit in [1] is the "Dynamic Array Tags" table, where it's stated that RUNPATH is optionally considered for both shared objects and executables, while RPATH is ignored for shared objects. It seems like that suggests the idea when using RUNPATH is that you'd have both executables and shared objects specifying it, instead of setting it just for the "top level" executable and letting that work transitively.

I've investigated this a bit and it seems that the move to using --enable-new-dtags by default is something Debian changed explicitly a while ago:

  binutils (2.27.51.20161116-2) unstable; urgency=medium

    * Stop building the mipsr6 mipsr6el mipsn32r6 mipsn32r6el mips64r6 mips64r6el
      variants; can't continue with this work, because package uploads with
      these architectures are still rejected.
    * Add homepage attribute to the control file: Closes: #841432.
    * ld: enable new dtags by default for linux/gnu targets. Closes: #835859.
    * Fix PR ld/20827, using proposed patch. Closes: #844378.

   -- Matthias Klose <email address hidden> Thu, 17 Nov 2016 11:56:55 +0100

While I understand this probably makes sense in a world where everyone is aware of these differences and either upstream projects are built considering RUNPATH (e.g. WebKitGTK+) or packagers explicitly workaround this by passing --disable-new-dtags to projects that haven't been adapted upstream yet (e.g. GNOME Shell), it still seems to me like this has huge potential to break things if it just gets enabled in other Debian-based without further consideration (and rebuilds!), so if you're having this problem and you control your platform but not every single package built on top of it, you might consider reverting the Debian patch that closes #835859 for the time being.

That's precisely what we're considering to do in Endless for now, and I thought I'd share it here in case this analysis was useful for someone else, if anything as a sign of gratitude for this bug report in itself, which was very useful for me to understand better what was going on when we first encountered this problem ourselves ...

[Javier Sixto (javyyer)]

This seems to be a feature of DT_RUNPATH that is do not follow transitive dependencies.
The problem is ld linker since v2.27(debian), enables by default new dtags ( it uses RUNPATH instead of RPATH), so some dependencies may not been found at runtime. Of course add LD_LIBRARY_PATH env, will do the tric, but those who use rpath options, are specially whiching to avoid that.

[Javier Sixto (javyyer)]

The problem comes from linker ld, which is in binutils.