Libraries from sandbox not found on execution

[Jan]

The modules' .so libraries are not found in the sandbox, e.g., when running the unit tests (see below).

When the LD_LIBRARY_PATH is set to the right sandbox lib directory, e.g. export LD_LIBRARY_PATH=/data/jan/openwns-sdk/sandbox/dbg/lib/ the unit tests run fine.

Without manual LD_LIBRARY_PATH setting:

jan@nepomuk:/data/jan/openwns-sdk/tests/unit/unitTests$ ./openwns -tv
Loading...
Library: ofdmaphy
Module: ofdmaphy

Loading libofdmaphy.so
Library: wifimac
Module: wifimac

Loading libwifimac.so
Library: ip
Module: ip

Loading libip.so
Library: constanze
Module: constanze

Loading libconstanze.so
Library: rise
Module: rise

Loading librise.so
Library: dllbase
Module: dll

Loading libdllbase.so
Library: simpletl
Module: simpleTL

Loading libsimpletl.so
ModuleFactory contains the following modules:
Backtrace (most recent call last, stack size: 10):
 10) __gxx_personality_v0
 9) __libc_start_main
 8) __gxx_personality_v0
 7) wns::simulator::IApplication::init()
 6) wns::simulator::Main<wns::simulator::Application>::doInit()
 5) wns::simulator::IApplication::init()
 4) wns::simulator::Application::doInit()
 3) wns::simulator::Application::loadModules()
 2) wns::Exception::Exception(std::string const&)
 1) wns::Backtrace::snapshot()
openWNS: Caught wns::Exception:

Can't load all specified modules. Reason:
libofdmaphy.so: cannot open shared object file: No such file or directorylibwifimac.so: cannot open shared object file: No such file or directorylibip.so: cannot open shared object file: No such file or directorylibconstanze.so: cannot open shared object file: No such file or directorylibrise.so: cannot open shared object file: No such file or directorylibdllbase.so: cannot open shared object file: No such file or directorylibsimpletl.so: cannot open shared object file: No such file or directory

[Karsten Klagges (karsten.klagges)]

please run an objdump on your openwns and post the result:

 # objdump -p openwns

[Jan (jan-ellenbeck)]

Here is the requested objdump. The missing shared object files are all located in the /data/jan/openwns-sdk/sandbox/dbg/lib directory that objdump shows. Is there any other information I should provide?

jan@nepomuk:/data/jan/openwns-sdk/tests/unit/unitTests$ objdump -p openwns

openwns: file format elf64-x86-64

Program Header:
    PHDR off 0x0000000000000040 vaddr 0x0000000000400040 paddr 0x0000000000400040 align 2**3
         filesz 0x00000000000001f8 memsz 0x00000000000001f8 flags r-x
  INTERP off 0x0000000000000238 vaddr 0x0000000000400238 paddr 0x0000000000400238 align 2**0
         filesz 0x000000000000001c memsz 0x000000000000001c flags r--
    LOAD off 0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**21
         filesz 0x0000000000003fb9 memsz 0x0000000000003fb9 flags r-x
    LOAD off 0x0000000000004d60 vaddr 0x0000000000604d60 paddr 0x0000000000604d60 align 2**21
         filesz 0x00000000000003b8 memsz 0x0000000000000708 flags rw-
 DYNAMIC off 0x0000000000004d90 vaddr 0x0000000000604d90 paddr 0x0000000000604d90 align 2**3
         filesz 0x0000000000000250 memsz 0x0000000000000250 flags rw-
    NOTE off 0x0000000000000254 vaddr 0x0000000000400254 paddr 0x0000000000400254 align 2**2
         filesz 0x0000000000000038 memsz 0x0000000000000038 flags r--
EH_FRAME off 0x0000000000003804 vaddr 0x0000000000403804 paddr 0x0000000000403804 align 2**2
         filesz 0x0000000000000114 memsz 0x0000000000000114 flags r--
   STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**3
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
   RELRO off 0x0000000000004d60 vaddr 0x0000000000604d60 paddr 0x0000000000604d60 align 2**0
         filesz 0x00000000000002a0 memsz 0x00000000000002a0 flags r--

Dynamic Section:
  NEEDED libboost_signals.so
  NEEDED libpython2.5.so.1.0
  NEEDED libwns.so
  NEEDED libboost_program_options.so
  NEEDED libboost_date_time.so
  NEEDED libcppunit-1.12.so.0
  NEEDED libstdc++.so.6
  NEEDED libm.so.6
  NEEDED libgcc_s.so.1
  NEEDED libc.so.6
  RPATH /data/jan/openwns-sdk/sandbox/dbg/lib
  RUNPATH /data/jan/openwns-sdk/sandbox/dbg/lib
  INIT 0x0000000000402180
  FINI 0x0000000000403558
  HASH 0x0000000000400290
  GNU_HASH 0x0000000000400518
  STRTAB 0x0000000000400fa8
  SYMTAB 0x00000000004006f0
  STRSZ 0x0000000000000cbb
  SYMENT 0x0000000000000018
  DEBUG 0x0000000000000000
  PLTGOT 0x0000000000604fe8
  PLTRELSZ 0x0000000000000318
  PLTREL 0x0000000000000007
  JMPREL 0x0000000000401e68
  RELA 0x0000000000401d90
  RELASZ 0x00000000000000d8
  RELAENT 0x0000000000000018
  VERNEED 0x0000000000401d20
  VERNEEDNUM 0x0000000000000003
  VERSYM 0x0000000000401c64

Version References:
  require...

[Jan (jan-ellenbeck)]

As suggsted offline by Karsten, I tried setting the"sandboxDir" option in config/options.py. This indeed makes him install the sandbox into a different directory. The the RPATH and RUNPATH entries are also updated to that new location. But still the libraries are not found there.

I then had a further look using strace to see where it looks for libraries. It turns out that normal system libraries (boost, cppunit etc) as listed under "NEEDED" above are indeed searched for in the sandbox directory as indicated by RPATH/RUNPATH. But the simulator looks for its own libraries (ofdmaphy, wns etc) only in standard library directories.openWNS's own shared libraries are only.

I guess the difference is that the simulator application itself is linked against the standard libraries and the rpath is honored. But the openWNS libraries are loaded using dlopen in library/module/Base.cpp

I wonder if the absolutePath feature in that function could help somehow. It would probably help on MacOS where loading libofdmaphy.so fails because the libraries are called xyz.dylib (e.g., libofdmaphy.dylib) but it wants to load a .so file.

libofdmaphy.so is only searched here:

open("tls/x86_64/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("tls/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("x86_64/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib64/tls/x86_64/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib64/tls/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib64/x86_64/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/lib64/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/x86_64/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/tls/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/x86_64/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib64/libofdmaphy.so", O_RDONLY) = -1 ENOENT (No such file or directory)
write(1, "Loading libofdmaphy.so\n", 23) = 23

Normal stuff is also searched in the sandbox:

pen("/data/jan/clean/sdk--main--1.0/sandbox/dbg/lib/tls/x86_64/libboost_signals.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/data/jan/clean/sdk--main--1.0/sandbox/dbg/lib/tls/x86_64", 0x7fff84c21270) = -1 ENOENT (No such file or directory)
open("/data/jan/clean/sdk--main--1.0/sandbox/dbg/lib/tls/libboost_signals.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/data/jan/clean/sdk--main--1.0/sandbox/dbg/lib/tls", 0x7fff84c21270) = -1 ENOENT (No such file or directory)
open("/data/jan/clean/sdk--main--1.0/sandbox/dbg/lib/x86_64/libboost_signals.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/data/jan/clean/sdk--main--1.0/sandbox/dbg/lib/x86_64", 0x7fff84c21270) = -1 ENOENT (No such file or directory)
open("/data/jan/clean/sdk--main--1.0/sandbox/dbg/lib/libboost_signals.so", O_RDONLY) = -1 ENOENT (No such file or directory)
stat("/data/jan/clean/sdk--main--1....

[Jan (jan-ellenbeck)]

OK, I found a fix for this and also an old SUSE bug report (from 2004) that seems to (re)appear on our openSUSE 11.0 system. Here is the old bug report: http://sources.redhat.com/ml/libc-alpha/2004-06/msg00116.html

It seems that if *both* RPATH and RUNPATH are set in the "Dynamic Section" of the openwns binary, the dlopen() call from library/module/Base.cpp ignores them completely and then fails to load the libraries. However, if only RPATH is set, it honors the setting and correctly loads the libraries.

To prevent the RUNPATH to be set, you can compile the main openwns application with (note the -Wl,-disable-new-dtags):

g++ -o .build/dbg/openwns-application/application--main--1.0/openwns '-Wl,-disable-new-dtags,-rpath=/data/jan/clean/openwns-sdk/sandbox/dbg/lib' .build/dbg/openwns-application/application--main--1.0/src/Main.o -Lsandbox/dbg/lib -lboost_signals -lpython2.5 -lwns -lboost_program_options -lboost_date_time -lcppunit

The resulting openws only has RPATH set:

jan@nepomuk:/data/jan/clean/openwns-sdk/tests/unit/unitTests$ objdump -p openwns-no-dtags
openwns-no-dtags: file format elf64-x86-64

Program Header:
    PHDR off 0x0000000000000040 vaddr 0x0000000000400040 paddr 0x0000000000400040 align 2**3
         filesz 0x00000000000001f8 memsz 0x00000000000001f8 flags r-x
  INTERP off 0x0000000000000238 vaddr 0x0000000000400238 paddr 0x0000000000400238 align 2**0
         filesz 0x000000000000001c memsz 0x000000000000001c flags r--
    LOAD off 0x0000000000000000 vaddr 0x0000000000400000 paddr 0x0000000000400000 align 2**21
         filesz 0x0000000000003fb9 memsz 0x0000000000003fb9 flags r-x
    LOAD off 0x0000000000004d70 vaddr 0x0000000000604d70 paddr 0x0000000000604d70 align 2**21
         filesz 0x00000000000003a8 memsz 0x00000000000006f8 flags rw-
 DYNAMIC off 0x0000000000004da0 vaddr 0x0000000000604da0 paddr 0x0000000000604da0 align 2**3
         filesz 0x0000000000000240 memsz 0x0000000000000240 flags rw-
    NOTE off 0x0000000000000254 vaddr 0x0000000000400254 paddr 0x0000000000400254 align 2**2
         filesz 0x0000000000000038 memsz 0x0000000000000038 flags r--
EH_FRAME off 0x0000000000003804 vaddr 0x0000000000403804 paddr 0x0000000000403804 align 2**2
         filesz 0x0000000000000114 memsz 0x0000000000000114 flags r--
   STACK off 0x0000000000000000 vaddr 0x0000000000000000 paddr 0x0000000000000000 align 2**3
         filesz 0x0000000000000000 memsz 0x0000000000000000 flags rw-
   RELRO off 0x0000000000004d70 vaddr 0x0000000000604d70 paddr 0x0000000000604d70 align 2**0
         filesz 0x0000000000000290 memsz 0x0000000000000290 flags r--

Dynamic Section:
  NEEDED libboost_signals.so
  NEEDED libpython2.5.so.1.0
  NEEDED libwns.so
  NEEDED libboost_program_options.so
  NEEDED libboost_date_time.so
  NEEDED libcppunit-1.12.so.0
  NEEDED libstdc++.so.6
  NEEDED libm.so.6
  NEEDED libgcc_s.so.1
  NEEDED libc.so.6
  RPATH /data/jan/clean/openwns-sdk/sandbox/dbg/lib
  INIT 0x0000000000402188
  FINI 0x00000000...

[Daniel Bültmann (daniel.bueltmann)]

Would you take care of this? Would we need to bring this back to main?

[Daniel Bültmann (daniel.bueltmann)]

Sorry. Got confused by launchpad user interface. I thought the last message would be sent to Karsten when I assigned him.

So:

Karsten, Would you take care of this? Would we need to bring this back to main?

Regards,
  Daniel

[Jan (jan-ellenbeck)]

I'm not sure if this bug just occurs on our special installation of openSUSE 11.0 or whether this is more generally related to a new gcc and linker version.

What I haven't figured out so far is how or where to include this "-disable-new-dtags" in the scons scripts to make the build process use it. I tried modifying the lines in framework/buildSupport/SConstruct where the "rpath" is set but that seems to be ignored. How would you add it?

[Maciej Muehleisen (mue-comnets)]

As far as I understand buildSupport is not used anymore. Everything
relevant is in the SConstcript in the openWNS root dir (used by all
modules through symlink), SConstruct in openWNS root dir and the special
SConscript in framework/application.

Jan wrote:
> I'm not sure if this bug just occurs on our special installation of
> openSUSE 11.0 or whether this is more generally related to a new gcc and
> linker version.
>
> What I haven't figured out so far is how or where to include this
> "-disable-new-dtags" in the scons scripts to make the build process use
> it. I tried modifying the lines in framework/buildSupport/SConstruct
> where the "rpath" is set but that seems to be ignored. How would you add
> it?
>
>

[Jan (jan-ellenbeck)]

That's an interesting insight - but how do you manage to set the rpath then? I grep'ed the whole openwns-sdk for "rpath" and it only shows up in that SConstruct inside buildSupport.

How about completely removing the buildSupport dir if it is not used anymore?

[Karsten Klagges (karsten.klagges)]

Well, the RPATH is set in the SConscript file of the openwns-application. And yes, removing the buildSupport is a good idea.

[Maciej Muehleisen (mue-comnets)]

RPATH is a scons Parameter and is therefore spelled upper case. I found
this line in framework/application/SConscript:
coreEnv.Append(RPATH = os.path.join(env.installDir, 'lib'))
I will test if everything works when buildSupport is removed and do it
if it works.

Jan wrote:
> That's an interesting insight - but how do you manage to set the rpath
> then? I grep'ed the whole openwns-sdk for "rpath" and it only shows up
> in that SConstruct inside buildSupport.
>
> How about completely removing the buildSupport dir if it is not used
> anymore?
>
>

[Jan (jan-ellenbeck)]

Finally, I can give a one-line bugfix for this. I just need to add this -disable-new-dtags option in the SConscript file in framework/application:
coreEnv.Append(LINKFLAGS = '-Wl,-disable-new-dtags')

Would you test this with your environment and include a patch to the mainline in case it works on every platform?

Cheers
Jan

Fix:
=== modified file 'SConscript'
--- SConscript 2008-10-30 22:18:33 +0000
+++ SConscript 2009-03-02 16:14:36 +0000
@@ -14,6 +14,9 @@
 if coreEnv['static']:
     LIBS += coreEnv['libraries']
     coreEnv.Append(LINKFLAGS = '-Wl,--whole-archive')
+
+coreEnv.Append(LINKFLAGS = '-Wl,-disable-new-dtags')
+
 LIBS = list(set(LIBS))
 prog = coreEnv.Program('openwns', files, LIBS = LIBS)
 coreEnv.Append(RPATH = os.path.join(env.installDir, 'lib'))

[Karsten Klagges (karsten.klagges)]

I have tested the option in the application SConscript and the build process seems to work. The man page of my ld says, the dtags are disabled by default. Could you give the version string of your ld and could you have a look at your manpage of ld? What daoes it say about the dtags. Are they enabled or disabled by default?

[Jan (jan-ellenbeck)]

jan@nepomuk:/data/jan/test/sdk--main--1.0$ ld --version
GNU ld (GNU Binutils; openSUSE 11.0) 2.18.50.20080409-11.1
Copyright 2007 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

Is this newer or older than yours?

On our system, it defaults to creating the new dtags, see below.

I think this is the vanilla ld coming with the openSUSE 11.0 x86-64

from "man ld"
[...]
--enable-new-dtags
--disable-new-dtags
           This linker can create the new dynamic tags in ELF. But the older ELF systems may not understand them. If you specify --enable-new-dtags, the dynamic tags will be created as needed. If you specify --disable-new-dtags, no new dynamic tags will be created. By default, the new dynamic tags are
           created. Note that those options are only available for ELF systems.

[Karsten Klagges (karsten.klagges)]

kks@erfurt:~$ ld --version
GNU ld (GNU Binutils for Ubuntu) 2.18.0.20080103
Copyright 2007 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

I will push the fix as soon as the test have finished successfully.