sprof fails to work with shared objects

[Indy S.]

Binary package hint: libc-dev-bin

When I try to profile a compiled shared object, sprof always fails with: "Inconsistency detected by ld.so: dl-open.c: 672: _dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed!"

I have the environment variables LD_PROFILE and LD_PROFILE_OUTPUT set up as you would expect and a file.so.profile file is created along with gmon.out.

Both the main program and .SO are compiled and linked with -pg

[Olivier (olivier-redhat-bugs)]

Created attachment 314122
Patch to add __RTLD_SPROF to dlopen_doit() mask

Description of problem:

In recent glibc such as the one included in RHEL5, sprof is not able to open shared librairies anymore. That used to work in previous versions of the glibc such as the one we shipped in RHEL4

Version-Release number of selected component (if applicable):

glibc-2.5-24

How reproducible:

100% reproducible

Steps to Reproduce:

1. LD_PROFILE_OUTPUT=${PWD} LD_PROFILE=libgtk-x11-2.0.so.0 gnome-calculator
2. sprof libgtk-x11-2.0.so.0
3.

Actual results:

sprof: failed to load shared object `libgtk-x11-2.0.so.0'

Expected results:

Flat profile:

Each sample counts as 0.01 seconds.
  % cumulative self self total
 time seconds seconds calls us/call us/call name

index % time self children called name

[0] 0.0 0.00 0.00 0 g_param_spec_object [0]
...

Additional info:

The bug was introduced in dlfcn/dlopen.c in dlopen_doit() by adding a mask for the mode and that mode is missing __RTLD_SPROF so that sprof cannot open the shared lib.

The patch attached adds __RTLD_SPROF to the mode mode so that sprof works.

[Ulrich (ulrich-redhat-bugs)]

Checked in upstream. Should be backported.

[RHEL (rhel-redhat-bugs)]

This request was evaluated by Red Hat Product Management for inclusion in a Red
Hat Enterprise Linux maintenance release. Product Management has requested
further review of this request by Red Hat Engineering, for potential
inclusion in a Red Hat Enterprise Linux Update release for currently deployed
products. This request is not yet committed for inclusion in an Update
release.

[Issue (issue-redhat-bugs)]

Another lead to follow up on. It might be that we can use some of the
userspace stap stuff. Evidently it is beginning to show up in 5.3.

Date: Fri, 17 Oct 2008 11:08:40 -0400
From: William Cohen <email address hidden>
User-Agent: Thunderbird 2.0.0.16 (X11/20080723)
MIME-Version: 1.0
To: Adam Jackson <email address hidden>
Subject: Re: RHEL Tools roadmap (internal use only for now)
References: <email address hidden>
        <email address hidden>
In-Reply-To: <email address hidden>
X-Scanned-By: MIMEDefang 2.58 on 172.16.27.26
X-Scanned-By: MIMEDefang 2.58 on 172.16.27.26
X-loop: <email address hidden>
Cc: tools <email address hidden>
X-BeenThere: <email address hidden>
X-Mailman-Version: 2.1.9
Precedence: list
List-Id: <tools.redhat.com>
List-Unsubscribe:
<http://post-office.corp.redhat.com/mailman/listinfo/tools>,
        <mailto:<email address hidden>?subject=unsubscribe>
List-Archive: <http://post-office.corp.redhat.com/archives/tools>
List-Post: <mailto:<email address hidden>>
List-Help: <mailto:<email address hidden>?subject=help>
List-Subscribe:
<http://post-office.corp.redhat.com/mailman/listinfo/tools>,
        <mailto:<email address hidden>?subject=subscribe>
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit
Sender: <email address hidden>
Errors-To: <email address hidden>

Adam Jackson wrote:
> On Thu, 2008-10-16 at 13:32 -0700, Eric Bachalo wrote:
>> Attached is latest draft of RHEL tools roadmap slides.
>> Any comments are welcome.
>
> I note userspace probing in systemtap in 5.3. Neat! A brief search on
> the systemtap wiki isn't turning up much about how to do this though.
> Got docs?

Hi Ajax,

You might use "man stapprobes" on with the systemtap-0.7.2-1.el5 in RHELU3
beta
for more information on user-space probes.

-Will

This event sent from IssueTracker by woodard
 issue 197199
it_file 169209

[errata-xmlrpc (errata-xmlrpc-redhat-bugs)]

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2009-0080.html

[Aidan Macdonald (aidan-plenert-macdonald)]

I am compiling without -pg and I still have this problem. Is there a work around yet?

[Ray Satiro (raysatiro)]

I am also having this problem. I'm trying to profile libcurl in Ubuntu 16.04 LTS and I get this:

export LD_PROFILE_OUTPUT=`pwd`
export LD_PROFILE=libcurl.so.4
src/.libs/lt-curl google.com > /dev/null
sprof lib/.libs/libcurl.so.4 libcurl.so.4.profile -p >log

_dl_open: Assertion `_dl_debug_initialize (0, args.nsid)->r_state == RT_CONSISTENT' failed!

I compiled with -fno-omit-frame-pointer -g -pg in CFLAGS and CXXFLAGS

[emakmel (gogo08)]

I am also having this problem, while trying to profile UBUNTU'S manual example.

Example found here : http://manpages.ubuntu.com/manpages/yakkety/man1/sprof.1.html#contenttoc7

Strace here:

execve("/usr/bin/sprof", ["sprof", "-p", "libdemo.so.1"], [/* 74 vars */]) = 0
brk(NULL) = 0x934000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0a39cd2000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=104012, ...}) = 0
mmap(NULL, 104012, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0a39cb8000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\r\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=14608, ...}) = 0
mmap(NULL, 2109680, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f0a398ab000
mprotect(0x7f0a398ae000, 2093056, PROT_NONE) = 0
mmap(0x7f0a39aad000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f0a39aad000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\t\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1868984, ...}) = 0
mmap(NULL, 3971488, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f0a394e1000
mprotect(0x7f0a396a1000, 2097152, PROT_NONE) = 0
mmap(0x7f0a398a1000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1c0000) = 0x7f0a398a1000
mmap(0x7f0a398a7000, 14752, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f0a398a7000
close(3) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0a39cb7000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0a39cb6000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0a39cb5000
arch_prctl(ARCH_SET_FS, 0x7f0a39cb6700) = 0
mprotect(0x7f0a398a1000, 16384, PROT_READ) = 0
mprotect(0x7f0a39aad000, 4096, PROT_READ) = 0
mprotect(0x604000, 4096, PROT_READ) = 0
mprotect(0x7f0a39cd4000, 4096, PROT_READ) = 0
munmap(0x7f0a39cb8000, 104012) = 0
brk(NULL) = 0x934000
brk(0x955000) = 0x955000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=10219008, ...}) = 0
mmap(NULL, 10219008, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0a38b22000
close(3) = 0
open("./libdemo.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\6\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0775, st_size=9496, ...}) = 0
getcwd("/home/keks/work/test", 128) = 21
mmap(N...