[armel/armhf] ltrace hangs

[Dr. David Alan Gilbert]

Binary package hint: ltrace

On armel ltrace just hangs:

ltrace /bin/ls
signal(55381, 0x00000001

ltrace: 0.5.3-2.1ubuntu1
kernel: 2.6.38-1207-omap4 #10-Ubuntu

This is running on a panda board.

Dave

[Tom Gall (tom-gall)]

using the developer build from 4/21 and the omap3 hwpack on my beagle Xm I see the same:

as root:
root@linaro:~# ltrace /bin/ls
signal(55377, 0x00000001Killed

as a mere mortal:
tgall@linaro:~/chromium$ ltrace /bin/ls
signal(55377, 0x00000001

root@linaro:~# dpkg -l ltrace
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-===============-===============-==============================================
ii ltrace 0.5.3-2.1ubuntu Tracks runtime library calls in dynamically li
root@linaro:~# uname -a
Linux linaro 2.6.38-1002-linaro-omap #3-Ubuntu SMP Fri Apr 15 14:00:54 UTC 2011 armv7l armv7l armv7l GNU/Linux
root@linaro:~#

[Riku Voipio (riku-voipio)]

Confirmed, Beagle B5, linary natty beta2 2.6.38-1002-linaro-omap

strace ltrace /bin/true
write(2, "__libc_start_main(", 18__libc_start_main() = 18
write(2, "36181", 536181) = 5
write(2, ", ", 2, ) = 2
write(2, "1", 11) = 1
write(2, ", ", 2, ) = 2
write(2, "0xbea048f4", 100xbea048f4) = 10
write(2, ", ", 2, ) = 2
write(2, "41393", 541393) = 5
write(2, ", ", 2, ) = 2
ptrace(PTRACE_PEEKDATA, 818, 0xbea048e8, [0xa1f1]) = 0
write(2, "41457", 541457) = 5
ptrace(PTRACE_PEEKTEXT, 818, 0x8b8e, [0xef0ef7ff]) = 0
ptrace(PTRACE_POKETEXT, 818, 0x8b8e, 0xef0ede01) = 0
ptrace(PTRACE_PEEKTEXT, 818, 0x89c4, [0xe7f001f0]) = 0
ptrace(PTRACE_POKETEXT, 818, 0x89c4, 0xe28fc600) = 0
ptrace(PTRACE_POKEUSER, 818, pc, 0x89c4) = 0
ptrace(PTRACE_SINGLESTEP, 818, 0, SIG_0) = -1 EIO (Input/output error)
wait4(-1,

One option is to ditch ltrace and use latrace instead.

[Dr. David Alan Gilbert (davidgil-uk)]

Indeed I've been using latrace, but I have a different selection of more subtle bugs with that.

Dave

[Riku Voipio (riku-voipio)]

We shall look at the tracing session at UDS if we should prefer ltrace or latrace. I think we should rather have one well working tool rather than two tools with different set of issues...

[Dave Martin (dave-martin-arm)]

I don't know whether this is the cause of the problem, but ltrace makes (incorrect) use of the PTRACE_SINGLESTEP ptrace() call.

Support for this ptrace call has been removed from the kernel as of v2.6.38-rc4, since it had few users, was already substantially broken for ARM and didn't support Thumb code at all.

If ltrace is still implicitly relying on this ptrace call working, it needs porting.

Cheers
---Dave

[Dave Martin (dave-martin-arm)]

For the kernel change, see

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=425fc47adb5bb69f76285be77a09a3341a30799e

[Dr. David Alan Gilbert (davidgil-uk)]

Interesting, well with that in hand the following patch seems to work OK;
I can see the MIPS sysdep has a separate implementation of set_instruction_pointer
that explicitly talks about avoiding PTRACE_SINGLESTEP but it's only MIPS that does that so far.

Dave

--- a/sysdeps/linux-gnu/trace.c
+++ b/sysdeps/linux-gnu/trace.c
@@ -127,7 +127,7 @@ continue_after_signal(pid_t pid, int signum) {

        proc = pid2proc(pid);
        if (proc && proc->breakpoint_being_enabled) {
-#if defined __sparc__ || defined __ia64___ || defined __mips__
+#if defined __sparc__ || defined __ia64___ || defined __mips__ || defined __arm__
                ptrace(PTRACE_SYSCALL, pid, 0, signum);
 #else
                ptrace(PTRACE_SINGLESTEP, pid, 0, signum);
@@ -162,7 +162,7 @@ continue_after_breakpoint(Process *proc, Breakpoint *sbp) {
        } else {
                debug(DEBUG_PROCESS, "continue_after_breakpoint: pid=%d, addr=%p", proc->pid, sbp->addr);
                proc->breakpoint_being_enabled = sbp;
-#if defined __sparc__ || defined __ia64___ || defined __mips__
+#if defined __sparc__ || defined __ia64___ || defined __mips__ || defined __arm__
                /* we don't want to singlestep here */
                continue_process(proc->pid);
 #else

[Dr. David Alan Gilbert (davidgil-uk)]

Oh, that patch is against git rev 4649a4cbb3bd6bc516c093adefa5dc7da8141b92 which still seems to be head-of-tree.

Dave

[Dr. David Alan Gilbert (davidgil-uk)]

ok, not so fast, so while that works for simple single processes, it fails three tests:

FAIL: sleep in /discs/more/daveg/git/ltrace/testsuite/ltrace.minor/attach-process.ltrace for 0 times, should be 1
FAIL: ^[0-9]* puts in /discs/more/daveg/git/ltrace/testsuite/ltrace.minor/trace-exec.ltrace for 0 times, should be 1
FAIL: ^[0-9]* __libc_start_main in /discs/more/daveg/git/ltrace/testsuite/ltrace.minor/trace-exec.ltrace for 1 times, should be 2

and it gets stuck or segfaults trying to run ltrace -f /bin/bash thistest

#!/bin/bash
/bin/ls
find . -name '*.c' -exec md5sum {} \;

[Avik Sil (aviksil)]

After applying the patch in comment #7 I could run ltrace successfully on Pandaboard and the testsuite passed too:

                === tests ===

Schedule of variations:
    unix

Running target unix
Using /usr/share/dejagnu/baseboards/unix.exp as board description file for target.
Using /usr/share/dejagnu/config/unix.exp as generic interface file for target.
Using ./config/unix.exp as tool-and-target-specific interface file.
Running ./ltrace.main/main-internal.exp ...
Running ./ltrace.main/main.exp ...
Running ./ltrace.main/parameters.exp ...
Running ./ltrace.main/signals.exp ...
Running ./ltrace.main/system_calls.exp ...
Running ./ltrace.minor/attach-process.exp ...
Running ./ltrace.minor/count-record.exp ...
Running ./ltrace.minor/demangle.exp ...
Running ./ltrace.minor/libdl-simple.exp ...
Running ./ltrace.minor/print-instruction-pointer.exp ...
Running ./ltrace.minor/time-record-T.exp ...
Running ./ltrace.minor/time-record-tt.exp ...
Running ./ltrace.minor/time-record-ttt.exp ...
Running ./ltrace.minor/trace-clone.exp ...
Running ./ltrace.minor/trace-exec.exp ...
Running ./ltrace.minor/trace-fork.exp ...
Running ./ltrace.torture/ia64-sigill.exp ...
Running ./ltrace.torture/signals.exp ...

                === Summary ===

# of expected passes 86
# of expected failures 2
make[4]: Leaving directory `/root/ltrace/testsuite'
make[3]: Leaving directory `/root/ltrace/testsuite'
make[2]: Leaving directory `/root/ltrace/testsuite'
make[1]: Leaving directory `/root/ltrace/testsuite'

[Dr. David Alan Gilbert (davidgil-uk)]

Hi Avik,
  Interesting - when I tried my patch it didn't pass the test suite (as described in comment 9) - which version of ltrace did you patch exactly?
  Does the 3 line shell script in patch 9 also work for you?

Dave

[Avik Sil (aviksil)]

Hi Dave,

I patched against the latest git rev 4649a4cbb3bd6bc516c093adefa5dc7da8141b92.

But when I ran the 3 line shell script I did get a segmentation fault.

Regards,
Avik

[Avik Sil (aviksil)]

With the final Linaro natty release, ltrace works fine with the above shell script: http://paste.ubuntu.com/622589/

[Will Newton (will-newton)]

ltrace in trusty hangs on ARM:

linaro@localhost:~$ ltrace /bin/ls
__libc_start_main(0x9ee9, 1, 0x7e9af7d4, 0x156b1PTRACE_SINGLESTEP: Input/output error
7943 couldn't continue when handling __libc_start_main (0x9c2c) at 0x9c2c

But upstream master works ok.

[Mathieu Trudel-Lapierre (cyphermox)]

I think we have enough information to land a fix for this. I've already reproduced on armhf on xenial with the exact same error:

root@ltrace:~# ltrace ls
__libc_start_main(0x12041, 1, 0x7ecd3894, 0x1e795PTRACE_SINGLESTEP: Input/output error
3872 couldn't continue when handling __libc_start_main (0x11d90) at 0x11d90

^C
^\Quit
root@ltrace:~# ltrace ps
__libc_start_main(0x11cbd, 1, 0x7ee16894, 0x19d01PTRACE_SINGLESTEP: Input/output error
3874 couldn't continue when handling __libc_start_main (0x11694) at 0x11694
^C^\Quit

And I've done some backporting of ltrace commits already. We'd just need to pinpoint which set of commits are still needed to fix that PTRACE_SINGLESTEP call on ARM. I'll track it down soonish.

[Geunsik Lim (leemgs)]

Um. I have also been getting sample problem on Ubuntu/ARM 14.04 as following:

root@target:~# ltrace
ltrace: too few arguments
Try `ltrace --help' for more information.
root@target:~# ltrace ls
__libc_start_main(0x9ee9, 1, 0x7ec70894, 0x156b1PTRACE_SINGLESTEP: Input/output error
1365 couldn't continue when handling __libc_start_main (0x9c2c) at 0x9c2c

[Geunsik Lim (leemgs)]

I have got the "PTRACE_SINGLESTEP" error message in case that I try to use ltrace 0.7.3 on Ubuntu/ARM 14.04.
Actually, This issue resulted from the change of the Linux Kernel.
Please, see the https://code.kodo.org.uk/dom/buildroot/commit/efee851c614926dd9ed0f49c4808a6d9b4f64eb0

[Heinrich Schuchardt (xypron)]

The Debian change was ingested into ltrace (0.7.3-6.1ubuntu1) disco. Mark the issue as "Fix Released".