gdb source test suites are failing in Ubuntu14.10

[bugproxy]

---Problem Description---
gdb source test suites are failing in Ubuntu14.10

Machine Type = P8

---Steps to Reproduce---
Install a P8 system with Power KVM and then install Ubuntu 14.10 guest.
Then try to build and execute the gdb source test suites as below.

root@ubuntu:~# apt-get source gdb
root@ubuntu:~# cd gdb-7.8/
root@ubuntu:~/gdb-7.8# dpkg-buildpackage -b 2>&1 | tee gdblog

                === gdb Summary ===

# of expected passes 25009
# of unexpected failures 287
# of unexpected successes 2
# of expected failures 66
# of unknown successes 1
# of known failures 59
# of unresolved testcases 3
# of untested testcases 19
# of unsupported tests 100

---uname output---
Linux ubuntu 3.16.0-11-generic #16-Ubuntu SMP Mon Aug 25 20:02:00 UTC 2014 ppc64le ppc64le ppc64le GNU/Linux

I've retested gdb from Ubuntu 14.04 (gdb-7.7-0ubuntu3.1) in the Ubuntu 14.10 VM provided by Pavaman and the number of failures curiously increased to 250 (in comparison to less than 140 on 14.04), which leads me to believe that there is something in the environment of Ubuntu 14.10 compromising gdb functionality in several testcases.

[bugproxy (bugproxy)]

Default Comment by Bridge

[Matthias Klose (doko)]

how did you check? are these regressions, or new tests failing?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-09-04 20:44 EDT-------
(In reply to comment #9)
> how did you check? are these regressions, or new tests failing?

These failures are mostly regressions. Here the failing tests that were OK on Ubuntu 14.04:

gdb.base/annota1.exp
gdb.base/annota3.exp
gdb.base/break-interp.exp
gdb.base/completion.exp
gdb.base/display.exp
gdb.base/ending-run.exp
gdb.base/shlib-call.exp
gdb.base/step-test.exp
gdb.base/store.exp
gdb.cp/koenig.exp
gdb.cp/operator.exp
gdb.cp/ovldbreak.exp
gdb.java/jprint.exp
gdb.mi/mi-stack.exp

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-09-04 20:53 EDT-------
In a quick test with gdb.base/annota3.exp testcase, all tests passed with gcc-4.8 and gcc-snapshot (there were 5 failures with default gcc-4.9).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-09-05 16:42 EDT-------
I reran gdb-7.8 testsuite on Ubuntu 14.10 using gcc-4.8 and the results seemed better:

=== gdb Summary ===

# of expected passes 25103
# of unexpected failures 190
# of unexpected successes 2
# of expected failures 66
# of unknown successes 1
# of known failures 59
# of unresolved testcases 3
# of untested testcases 18
# of unsupported tests 100

For comparison reasons, this is gdb-7.7 testsuite from Ubuntu 14.04 results on Ubuntu 14.10 with gcc-4.8:

=== gdb Summary ===

# of expected passes 22321
# of unexpected failures 155
# of unexpected successes 2
# of expected failures 65
# of unknown successes 1
# of known failures 59
# of unresolved testcases 6
# of untested testcases 20
# of unsupported tests 67

In this case, the difference of unexpected failures between them are due to new testcases (mostly gdb.base/break-idempotent.exp and gdb.python/py-xmethods.exp).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-09-05 16:51 EDT-------
FWIW, I've also rerun the tests using gcc-snapshot (gcc-5.0.0) on Ubuntu 14.10. The results look better than gcc-4.9 but no as good as gcc-4.8:

gdb-7.7 (from Ubuntu 14.04):

=== gdb Summary ===

# of expected passes 22364
# of unexpected failures 167
# of unexpected successes 1
# of expected failures 64
# of known failures 56
# of unresolved testcases 6
# of untested testcases 19
# of unsupported tests 66

gdb-7.8 (from Ubuntu 14.10):

=== gdb Summary ===

# of expected passes 25138
# of unexpected failures 210
# of unexpected successes 1
# of expected failures 65
# of known failures 56
# of unresolved testcases 2
# of untested testcases 17
# of unsupported tests 104

[Anton Blanchard (anton-samba)]

(gdb) break /home/ubuntu/binutils-gdb/gdb/testsuite/gdb.base/annota1.c:28
Breakpoint 1 at 0x100006e4: file /home/ubuntu/binutils-gdb/gdb/testsuite/gdb.base/annota1.c, line 28.

(gdb) run
Starting program: /tmp/foo
Can't read symbols from system-supplied DSO at 0x3fffb7fa0000: File truncated

Breakpoint 1, main () at /home/ubuntu/binutils-gdb/gdb/testsuite/gdb.base/annota1.c:28
28 int my_array[3] = { 1, 2, 3 }; /* break main */

I wonder if the gdb complaint about the kernel VDSO is causing the test to fail. The warning is almost certainly a result of
this kernel patch:

commit 24b659a13866b935eca72748ce725279bd3c4466
Author: Anton Blanchard <email address hidden>
Date: Wed Feb 12 17:18:50 2014 +1100

    powerpc: Use unstripped VDSO image for more accurate profiling data

    We are seeing a lot of hits in the VDSO that are not resolved by perf.
    A while(1) gettimeofday() loop shows the issue:

    27.64% [vdso] [.] 0x000000000000060c
    22.57% [vdso] [.] 0x0000000000000628
    16.88% [vdso] [.] 0x0000000000000610
    12.39% [vdso] [.] __kernel_gettimeofday
     6.09% [vdso] [.] 0x00000000000005f8
     3.58% test [.] 00000037.plt_call.gettimeofday@@GLIBC_2.18
     2.94% [vdso] [.] __kernel_datapage_offset
     2.90% test [.] main

    We are using a stripped VDSO image which means only symbols with
    relocation info can be resolved. There isn't a lot of point to
    stripping the VDSO, the debug info is only about 1kB:

    4680 arch/powerpc/kernel/vdso64/vdso64.so
    5815 arch/powerpc/kernel/vdso64/vdso64.so.dbg

    By using the unstripped image, we can resolve all the symbols in the
    VDSO and the perf profile data looks much better:

    76.53% [vdso] [.] __do_get_tspec
    12.20% [vdso] [.] __kernel_gettimeofday
     5.05% [vdso] [.] __get_datapage
     3.20% test [.] main
     2.92% test [.] 00000037.plt_call.gettimeofday@@GLIBC_2.18

[Ulrich Weigand (uweigand)]

Anton Blanchard wrote:

>I wonder if the gdb complaint about the kernel VDSO is causing the test to fail. The warning is almost certainly a result of
>this kernel patch:
>
> powerpc: Use unstripped VDSO image for more accurate profiling data

Whether or not this complaint is causing that test to fail, it should be fixed.

Using an unstripped VDSO image is certainly better than a stripped one, also for debugging purposes. However, it needs to be correct (and complete) -- maybe the VDSO is indeed truncated? Including full debug data, the VDSO may be longer than a page, does the kernel actually map all of it to user space?

[Anton Blanchard (anton-samba)]

I see what is going on, but don't know how to fix it yet.

Gdb must be checking for program headers that are marked load or segments that are marked allocate (not exactly sure what yet). The .symtab and .strtab are not, because normally they are not loaded into memory.

The problem is bfd_elf_get_elf_syms tries to bfd_seek too far, and fails. The VDSO looks like:

[Nr] Name Type Address Off Size ES Flg Lk Inf Al
  [ 0] NULL 0000000000000000 000000 000000 00 0 0 0
  [ 1] .hash HASH 0000000000000120 000120 00004c 04 A 2 0 8
  [ 2] .dynsym DYNSYM 0000000000000170 000170 000150 18 A 3 2 8
  [ 3] .dynstr STRTAB 00000000000002c0 0002c0 00010d 00 A 0 0 1
  [ 4] .gnu.version VERSYM 00000000000003ce 0003ce 00001c 02 A 2 0 2
  [ 5] .gnu.version_d VERDEF 00000000000003f0 0003f0 000038 00 A 3 2 8
  [ 6] .note NOTE 0000000000000428 000428 00003c 00 A 0 0 4
  [ 7] .text PROGBITS 0000000000000470 000470 000324 00 AX 0 0 8
  [ 8] .dynamic DYNAMIC 0000000000000798 000798 000100 10 WA 3 0 8
  [ 9] .eh_frame_hdr PROGBITS 0000000000000898 000898 00006c 00 A 0 0 4
  [10] .eh_frame PROGBITS 0000000000000908 000908 000538 00 A 0 0 8
  [11] .got PROGBITS 0000000000000e40 000e40 000008 08 WA 0 0 8
  [12] .debug_aranges PROGBITS 0000000000000000 000e50 0000f0 00 0 0 16
  [13] .debug_info PROGBITS 0000000000000000 000f40 00034c 00 0 0 1
  [14] .debug_abbrev PROGBITS 0000000000000000 00128c 000064 00 0 0 1
  [15] .debug_line PROGBITS 0000000000000000 0012f0 000313 00 0 0 1
  [16] .rela.dyn RELA 0000000000000e40 000e40 000000 18 A 2 0 8
  [17] .shstrtab STRTAB 0000000000000000 001603 0000be 00 0 0 1
  [18] .symtab SYMTAB 0000000000000000 001bc8 000318 18 19 21 8
  [19] .strtab STRTAB 0000000000000000 001ee0 000127 00 0 0 1

strace output:

open("/proc/19509/mem", O_RDONLY|O_CLOEXEC) = 10
pread64(10, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\25\0\1\0\0\0p\4\0\0\0\0\0\0"..., 64, 70367535824896) = 64
close(10) = 0
open("/proc/19509/mem", O_RDONLY|O_CLOEXEC) = 10
pread64(10, "\1\0\0\0\5\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 224, 70367535824960) = 224
close(10) = 0
open("/proc/19509/mem", O_RDONLY|O_CLOEXEC) = 10
pread64(10, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0\25\0\1\0\0\0p\4\0\0\0\0\0\0"..., 7112, 70367535824896) = 7112
close(10) = 0

We read up to 7112 (0x1bc8), right up to before the symtab. Not sure it reads to there and not the limit of the program load header or all section headers marked allocate.

[Ulrich Weigand (uweigand)]

Well, GDB assumes that the VDSO contains the whole ELF *file*, not just segments that would be loaded into memory if the object were loaded by the dynamic loader.

In particular, none of the symbol table (.symtab, .strtab) or debug info sections usually reside in loaded segments, but the debugger of course has to access them.

When debugging "normal" executables or shared librares, GDB will just read that information from the file on disk. But for a VSDO, no such file exists; instead, GDB assumes the full file is actually present in memory.

("Full file" here means everything covered by any *section*, not just those that are part of a loaded segment.)

[Anton Blanchard (anton-samba)]

Still trawling through gdb, but the issue appears to be in bfd/elfcode.h, bfd_from_remote_memory:

shdr_end = i_ehdr.e_shoff + i_ehdr.e_shnum * i_ehdr.e_shentsize;

  [17] .shstrtab STRTAB 0000000000000000 001603 0000be 00 0 0 1
  [18] .symtab SYMTAB 0000000000000000 001bc8 000318 18 19 21 8

i_ehdr.e_shoff = 0x16c8. What is in the gap between 0x16c8 and 0x1bc8? Do we have a VDSO linker script issue? (entire elf segment listing is in comment #9)

[Anton Blanchard (anton-samba)]

Of course it's the actual section headers at that offset. Now the question is, should we fix bfd_from_remote_memory so it can read sections after the section headers assuming they are in memory.

[Ulrich Weigand (uweigand)]

Hmmm ... usually, the section headers come at the very end of the file. It's a bit strange that there is section data *after* the headers in this case; maybe that's what confuses GDB/BFD.

[Ulrich Weigand (uweigand)]

On second glance, looks I'm wrong: .symtab and .strtab actually do come *after* the section headers in "normal" executables too.
So I guess the best thing to do would be to fix bfd_from_remote_memory to actually include enough memory to cover all sections, even those after the section headers. (Assuming they are actually included in the in-memory size of the VDSO, which bfd_from_remote_memory actually knows.)

[Alan Modra (amodra)]

I'm looking into moving the section headers to the end of the object file for ld.bfd. It doesn't look too hard, but there will likely be some testsuite tweaking needed. It looks like gold puts section headers last.

[Ulrich Weigand (uweigand)]

Moving section headers to the end sounds good.

However, maybe we should make the bfd_from_remote_memory change too, in order to cope with kernels already in the field?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-09-30 17:10 EDT-------
This issue with the DSO warning message was fixed recently on upstream:

https://sourceware.org/ml/gdb-patches/2014-09/msg00594.html

I've rebuild gdb 7.8-0ubuntu1 with the aforementioned patch using gcc-4.9.1, it indeed fixes the DSO issue and the number of unexpected failures was reduced a bit:

=== gdb Summary ===

# of expected passes 24308
# of unexpected failures 254
# of unexpected successes 2
# of expected failures 57
# of unknown successes 1
# of known failures 58
# of unresolved testcases 4
# of untested testcases 40
# of unsupported tests 191

However, the number of failures is still higher than when using gcc-4.8.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2014-10-01 16:45 EDT-------
FYI, I rebuilt gdb packages including the aforementioned patch and uploaded them to:

http://ausgsa.ibm.com/~emachado/public/gdb/lp13656641/

[Launchpad Janitor (janitor)]

This bug was fixed in the package gdb - 7.8-0ubuntu2

---------------
gdb (7.8-0ubuntu2) utopic; urgency=medium

  * Update gdb from the 7.8 release branch.
    - Fix regression for Linux vDSO in GDB (PR gdb/17407).
    - Fix regression, GDB stopped on run with attached process (PR gdb/17347).
    - Fix PR guile/17367, PR guile/17247.
    - Fix crash on Python frame filters with unreadable arg.
    - Fix xmethod Python so that it works with Python3.
    - Fix 'gcore' with exited threads.
    - Fix Fix build/17104, build configured with --with-babeltrace.
    - Aarch64: Make CPSR a 32-bit register again in the target description.
  * Build-depend on binutils 2.24.51.20141001 for running the tests.
    LP: #1365664.
  * Fix gdbserver conditionals on ppc64le. LP: #1367832.
  * Fix attaching to a java process on ppc64el.
 -- Matthias Klose <email address hidden> Wed, 01 Oct 2014 22:41:51 +0200

[Matthias Klose (doko)]

binutils is now recent enough in 14.10