imacros command line breaks debugging information

[pixel]

Hello,

  Currently, the gcc ubuntu package has a bug that comes from upstream:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=57653

  The effect of the bug is if one compiles a file with the option -imacros,
the filename and all useful debugging information disappears. Here is a
simple example of that:

imacros-test$ cat somefile.c
#warning some warning
imacros-test$ cat macros.h
#define SOME_MACRO
imacros-test$ gcc -c somefile.c
somefile.c:1:2: warning: #warning some warning [-Wcpp]
 #warning some warning
  ^
imacros-test$ gcc -c somefile.c -imacros macros.h
<command-line>:1:2: warning: #warning some warning [-Wcpp]

  This also affects the debugging information embedded within the files,
rendering debugging pretty useless.

  While I am aware this is a bug from upstream, it's been sitting there
for a while with a simple patch. It'd be nice to apply that patch on the
ubuntu package in the meantime.

  Thanks!

ProblemType: Bug
DistroRelease: Ubuntu 13.10
Package: gcc 4:4.8.1-2ubuntu3
ProcVersionSignature: Ubuntu 3.8.0-19.30-generic 3.8.8
Uname: Linux 3.8.0-19-generic x86_64
ApportVersion: 2.12.5-0ubuntu2.2
Architecture: amd64
Date: Mon Jan 6 05:41:13 2014
MarkForUpload: True
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: gcc-defaults
UpgradeStatus: Upgraded to saucy on 2013-10-19 (78 days ago)

[Allan McRae (allan-archlinux)]

When compiling with the "-imacros" flag, the file name information is lost in error messages and backtraces.

e.g.

> gcc-4.8 -imacros i.h test.c
<command-line>: In function ‘main’:
<command-line>:5:1: error: expected ‘;’ before ‘}’ token

The problem commit is:
http://gcc.gnu.org/git/?p=gcc.git;a=commitdiff;h=6adc88f8

[Manu-gcc (manu-gcc)]

Can you provide a complete testcase? Thanks.

[Allan McRae (allan-archlinux)]

# echo "int main() { return }" > foo.c
# touch foo.h
# gcc -imacros foo.h foo.c
<command-line>: In function ‘main’:
<command-line>:1:21: error: expected expression before ‘}’ token

[Manu-gcc (manu-gcc)]

(In reply to Allan McRae from comment #2)
> # echo "int main() { return }" > foo.c
> # touch foo.h
> # gcc -imacros foo.h foo.c
> <command-line>: In function ‘main’:
> <command-line>:1:21: error: expected expression before ‘}’ token

I cannot reproduce this with r198545, so it seems fixed in mainline. Could you test with that?

[Manu-gcc (manu-gcc)]

I mean in GCC 4.9.0 (trunk), (that revision is the latest I have built).

[Allan McRae (allan-archlinux)]

I still get this with r200224

# gcc-4.9 -v
Using built-in specs.
COLLECT_GCC=gcc-4.9
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-unknown-linux-gnu/4.9.0/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: /build/gcc-git/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/share/gcc-4.9 --with-bugurl='http://aur.archlinux.org/packages.php?ID=16045' --enable-languages=c,c++,lto --enable-shared --enable-threads=posix --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --enable-gnu-unique-object --enable-linker-build-id --enable-cloog-backend=isl --disable-cloog-version-check --enable-lto --enable-gold --enable-ld=default --enable-plugin --with-plugin-ld=ld.gold --with-linker-hash-style=gnu --disable-install-libiberty --disable-multilib --disable-libssp --disable-werror --enable-checking=release --program-suffix=-4.9 --enable-version-specific-runtime-libs
Thread model: posix
gcc version 4.9.0 20130619 (experimental) (GCC)

Built with

CFLAGS="-march=x86-64 -mtune=generic -O2 -fstack-protector --param=ssp-buffer-size=4"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro"

[Allan McRae (allan-archlinux)]

I have also confirmed this issue on Fedora rawhide.

gcc 4.8.1 20130603 (Red Hat 4.8.1-1)

[Manu-gcc (manu-gcc)]

(In reply to Allan McRae from comment #6)
> I have also confirmed this issue on Fedora rawhide.
>
> gcc 4.8.1 20130603 (Red Hat 4.8.1-1)

What I would do to investigate this issue is to put a breakpoint in linemap_add and try to figure out what is the difference with and without the -imacro. There must be some difference there.

[Allan McRae (allan-archlinux)]

I really have no idea what I am looking for... but adding a breakpoint at linemap_add I see (reason, file):

LC_ENTER "foo.c"
LC_RENAME "<command-line>"
LC_ENTER "/usr/include/stdc-predef.h"
LC_LEAVE 0x0
LC_RENAME "foo.c"
<- correct output printed here
LC_LEAVE 0x0

LC_ENTER "foo.c"
LC_RENAME "<command-line>"
LC_ENTER "foo.h"
LC_LEAVE 0x0
LC_ENTER "/usr/include/stdc-predef.h"
LC_RENAME "foo.c"
LC_LEAVE 0x0
<- incorrect output printed here
LC_LEAVE 0x0

So it looks like it is not leaving "/usr/include/stdc-predef.h" at the right time so the wrong thing is being renamed?

[Manu-gcc (manu-gcc)]

(In reply to Allan McRae from comment #8)
> I really have no idea what I am looking for... but adding a breakpoint at
> linemap_add I see (reason, file):
>
> LC_ENTER "foo.c"
> LC_RENAME "<command-line>"
> LC_ENTER "/usr/include/stdc-predef.h"
> LC_LEAVE 0x0
> LC_RENAME "foo.c"
> <- correct output printed here
> LC_LEAVE 0x0
>
> LC_ENTER "foo.c"
> LC_RENAME "<command-line>"
> LC_ENTER "foo.h"
> LC_LEAVE 0x0
> LC_ENTER "/usr/include/stdc-predef.h"
> LC_RENAME "foo.c"
> LC_LEAVE 0x0
> <- incorrect output printed here
> LC_LEAVE 0x0
>
> So it looks like it is not leaving "/usr/include/stdc-predef.h" at the right
> time so the wrong thing is being renamed?

It seems a probable cause for this bug. Can you track where the return paths differ?

Does

LC_ENTER "/usr/include/stdc-predef.h"

happen through the same functions in both cases?

and the LC_LEAVE 0x0 just afterwards?

Perhaps there is a missing LC_LEAVE or perhaps the order between

 LC_RENAME "foo.c"
 LC_LEAVE 0x0

should be reversed.

Strangely, I still cannot reproduce this:

manuel@gcc10:~$ ~/test1/200330/install/bin/gcc -imacros foo.h foo.c
foo.c: In function ‘main’:
foo.c:1:21: error: expected expression before ‘}’ token
 int main() { return }
                     ^
manuel@gcc10:~$ ~/test1/200330/install/bin/gcc -v
Using built-in specs.
COLLECT_GCC=/home/manuel/test1/200330/install/bin/gcc
COLLECT_LTO_WRAPPER=/home/manuel/test1/200330/install/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.9.0/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: /home/manuel/test1/src/configure --prefix=/home/manuel/test1/./200330/install --enable-languages=c,c++,objc,fortran,ada,obj-c++
Thread model: posix
gcc version 4.9.0 20130622 (experimental) [trunk revision 195333] (GCC)

Are you sure you don't have some local arch-linux patches? Have you tried directly downloading the svn version?

[Manu-gcc (manu-gcc)]

Sorry, I didn't intend to confirm it until someone else can reproduce it.

[Allan McRae (allan-archlinux)]

Created attachment 30345
log of gdb session

Here is the log from my gdb session.

Arch builds with no patches, just a could of small sed lines that should not affect this. See our build script (plain bash):
https://projects.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/gcc

Since it is also in Fedora rawhide, hopefully someone there can confirm.

[Manu-gcc (manu-gcc)]

(In reply to Allan McRae from comment #11)
> Created attachment 30345 [details]
> log of gdb session
>
> Here is the log from my gdb session.
>
> Arch builds with no patches, just a could of small sed lines that should not
> affect this. See our build script (plain bash):
> https://projects.archlinux.org/svntogit/packages.git/tree/trunk/
> PKGBUILD?h=packages/gcc

Does it do a full bootstrap? My gdb session looks more optimized than yours. Perhaps it is a bug in the host compiler?

My version also does not ever execute this:

Breakpoint 1, linemap_add (set=0x7ffff7ff9000, reason=LC_ENTER, sysp=2,
    to_file=0x140e270 "/usr/include/stdc-predef.h", to_line=1)
    at /build/gcc-git/src/gcc/libcpp/line-map.c:291

In fact, I don't have this file.
manuel@gcc10:~$ cat /etc/debian_version
6.0.6

so the presence of the file may make a difference (unfortunately, I don't have root access to gcc10 to fake the file and test).

[Allan McRae (allan-archlinux)]

The Arch gcc does the full bootstrap. The debug build I am using was compiled with DEBUG_CFLAGS="-g -fvar-tracking-assignments".

The file "/usr/include/stdc-predef.h" is from glibc (v2.17 on Arch) and is specifically mentioned as being preincluded in http://gcc.gnu.org/gcc-4.8/porting_to.html. In fact, using -ffreestanding "solves" the issue.

[Manu-gcc (manu-gcc)]

(In reply to Allan McRae from comment #13)
> The file "/usr/include/stdc-predef.h" is from glibc (v2.17 on Arch) and is
> specifically mentioned as being preincluded in
> http://gcc.gnu.org/gcc-4.8/porting_to.html. In fact, using -ffreestanding
> "solves" the issue.

So when you use -ffreestanding, is stdc-predef.h still included?

You could put a break in push_command_line_include and check if -include foo.h still includes stdc-predef.h and whether it shows also the problem. If the file is included but there is no bug, then my guess is that the code executed before or after the pre-include stdc-predef.h is missing something for the -imacros case. If it shows the bug, then the code for pre-including stuff must be wrong somehow, perhaps cpp_push_default_include is doing something wrong when compared to cpp_push_include.

[Allan McRae (allan-archlinux)]

with -ffreestanding

LC_ENTER "foo.c"
LC_RENAME "<command-line>"
LC_ENTER "foo.h"
LC_LEAVE 0x0
LC_RENAME "foo.c"
<- correct output printed here
LC_LEAVE 0x0

so std-predef.h is not included.

I tried with -include foo.h and a breakpoint in push_command_line_include... I just stepped through from that breakpoint and saw cpp_push_default_include being called for both stdc-predef.h and foo.h. That also gives the correct output.

# gcc-4.9 -include foo.h foo.c
foo.c: In function ‘main’:
foo.c:1:21: error: expected expression before ‘}’ token
 int main() { return }

[Manu-gcc (manu-gcc)]

(In reply to Allan McRae from comment #15)
>
> I tried with -include foo.h and a breakpoint in push_command_line_include...
> I just stepped through from that breakpoint and saw cpp_push_default_include
> being called for both stdc-predef.h and foo.h. That also gives the correct
> output.
>
> # gcc-4.9 -include foo.h foo.c
> foo.c: In function ‘main’:
> foo.c:1:21: error: expected expression before ‘}’ token
> int main() { return }

What is the trace of LC_ messages produced here? It seems the -imacros code is missing something. Actually, if you could attach the corresponding gdb session like you did earlier, it would be helpful to compare both.

[Allan McRae (allan-archlinux)]

Created attachment 30359
gdb log when using -include

When using -include instead of -imacros, the trail is:

LC_ENTER foo.c
LC_RENAME <command-line>
LC_ENTER /usr/include/stdc-predef.h
LC_LEAVE 0x0
LC_ENTER foo.h
LC_LEAVE 0x0
LC_RENAME foo.c
LC_LEAVE 0x0

gdb log with -include attached.

[Manu-gcc (manu-gcc)]

Thanks!

I think I know the reason why it is failing, but I am not sure about the proper fix.

For some reason unknown to me, push_commandline_include should not be called while processing -imacros. -imacros tries to achieve this but playing tricks with include_cursor, but that doesn't stop the pre-included files. Calling cpp_push_include (or cpp_push_default_include) seems to mess up everything (again, no idea why!). This code is really a mess but the simple patch below seems to work. Could you test it?

Index: gcc/c-family/c-opts.c
===================================================================
--- gcc/c-family/c-opts.c (revision 200330)
+++ gcc/c-family/c-opts.c (working copy)
@@ -1338,10 +1338,14 @@ c_finish_options (void)

 /* Give CPP the next file given by -include, if any. */
 static void
 push_command_line_include (void)
 {
+ // This can happen if disabled by -imacros for example.
+ if (include_cursor > deferred_count)
+ return;
+
   if (!done_preinclude)
     {
       done_preinclude = true;
       if (flag_hosted && std_inc && !cpp_opts->preprocessed)
        {

[Allan McRae (allan-archlinux)]

That patch works. With -imacros foo.h:

LC_ENTER foo.c
LC_RENAME <command-line>
LC_ENTER foo.h
LC_LEAVE 0x0
LC_ENTER /usr/include/stdc-predef.h
LC_LEAVE 0x0
LC_RENAME foo.c
<- correct output
LC_LEAVE 0x0

All other combinations of -include/-imacros/-ffreestanding/no foo.h all are unchanged as expected.

[Manu-gcc (manu-gcc)]

(In reply to Allan McRae from comment #19)
> That patch works. With -imacros foo.h:
>
> LC_ENTER foo.c
> LC_RENAME <command-line>
> LC_ENTER foo.h
> LC_LEAVE 0x0
> LC_ENTER /usr/include/stdc-predef.h
> LC_LEAVE 0x0
> LC_RENAME foo.c
> <- correct output
> LC_LEAVE 0x0
>
> All other combinations of -include/-imacros/-ffreestanding/no foo.h all are
> unchanged as expected.

Great! I am certainly too busy at the moment to go through all the steps of the contribution process for this. The patch needs to be fully bootstrapped with the latest trunk, an appropriate testcase needs to be created, plus run the full regression testsuite and compare the results with the regression results of the unpatched trunk and check that no new failures appear. Then, it has to be submitted to gcc-patches with a GNU Changelog.

If you are really interested in fixing this, you should just take the lead. Feel free to modify the patch as you need. The patch is anyway too small to require any kind of legal assignment, but if they ask for it, you can always say I wrote it. I have an assignment in place with the FSF.

When submitting to gcc-patches, CC <email address hidden>. You can also see in the svn log or svn blame who wrote the -imacro code and CC him/her as well. (You can also CC me).

[Manu-gcc (manu-gcc)]

Once you are in trunk, you can ask the release managers to backport it to the GCC 4.8 branch.

[Mpolacek (mpolacek)]

Author: mpolacek
Date: Thu Jul 24 09:00:13 2014
New Revision: 212972

URL: https://gcc.gnu.org/viewcvs?rev=212972&root=gcc&view=rev
Log:
 PR c/57653
 * c-opts.c (c_finish_options): If -imacros is in effect, return.

 * c-c++-common/pr57653.c: New test.
 * c-c++-common/pr57653.h: New file.
 * c-c++-common/pr57653-2.c: New test.
 * c-c++-common/pr57653-2.h: New file.

Added:
    trunk/gcc/testsuite/c-c++-common/pr57653-2.c
    trunk/gcc/testsuite/c-c++-common/pr57653-2.h
    trunk/gcc/testsuite/c-c++-common/pr57653.c
    trunk/gcc/testsuite/c-c++-common/pr57653.h
Modified:
    trunk/gcc/c-family/ChangeLog
    trunk/gcc/c-family/c-opts.c
    trunk/gcc/testsuite/ChangeLog

[Rguenth (rguenth)]

If 4.7.3 works this is a regression. Can we get the patch backported to release branches please?

[Rguenth (rguenth)]

Anyway, I'll do that (tested 4.8 already).

[Rguenth (rguenth)]

Author: rguenth
Date: Wed Feb 11 12:14:07 2015
New Revision: 220614

URL: https://gcc.gnu.org/viewcvs?rev=220614&root=gcc&view=rev
Log:
2015-02-11 Richard Biener <email address hidden>

 Backport from mainline
 2014-07-24 Marek Polacek <email address hidden>

 PR c/57653
 * c-opts.c (c_finish_options): If -imacros is in effect, return.

 * c-c++-common/pr57653.c: New test.
 * c-c++-common/pr57653.h: New file.
 * c-c++-common/pr57653-2.c: New test.
 * c-c++-common/pr57653-2.h: New file.

Added:
    branches/gcc-4_9-branch/gcc/testsuite/c-c++-common/pr57653-2.c
    branches/gcc-4_9-branch/gcc/testsuite/c-c++-common/pr57653-2.h
    branches/gcc-4_9-branch/gcc/testsuite/c-c++-common/pr57653.c
    branches/gcc-4_9-branch/gcc/testsuite/c-c++-common/pr57653.h
Modified:
    branches/gcc-4_9-branch/gcc/c-family/ChangeLog
    branches/gcc-4_9-branch/gcc/c-family/c-opts.c
    branches/gcc-4_9-branch/gcc/testsuite/ChangeLog

[Rguenth (rguenth)]

Author: rguenth
Date: Wed Feb 11 12:14:54 2015
New Revision: 220615

URL: https://gcc.gnu.org/viewcvs?rev=220615&root=gcc&view=rev
Log:
2015-02-11 Richard Biener <email address hidden>

 Backport from mainline
 2014-07-24 Marek Polacek <email address hidden>

 PR c/57653
 * c-opts.c (c_finish_options): If -imacros is in effect, return.

 * c-c++-common/pr57653.c: New test.
 * c-c++-common/pr57653.h: New file.
 * c-c++-common/pr57653-2.c: New test.
 * c-c++-common/pr57653-2.h: New file.

Added:
    branches/gcc-4_8-branch/gcc/testsuite/c-c++-common/pr57653-2.c
    branches/gcc-4_8-branch/gcc/testsuite/c-c++-common/pr57653-2.h
    branches/gcc-4_8-branch/gcc/testsuite/c-c++-common/pr57653.c
    branches/gcc-4_8-branch/gcc/testsuite/c-c++-common/pr57653.h
Modified:
    branches/gcc-4_8-branch/gcc/c-family/c-opts.c
    branches/gcc-4_8-branch/gcc/testsuite/ChangeLog

[Rguenth (rguenth)]

Fixed.

[Rguenth (rguenth)]

Author: rguenth
Date: Thu Feb 12 09:48:56 2015
New Revision: 220642

URL: https://gcc.gnu.org/viewcvs?rev=220642&root=gcc&view=rev
Log:
2015-02-11 Richard Biener <email address hidden>

 Backport from mainline
 2014-07-24 Marek Polacek <email address hidden>

 PR c/57653
 * c-opts.c (c_finish_options): If -imacros is in effect, return.

Modified:
    branches/gcc-4_8-branch/gcc/c-family/ChangeLog

[Matthias Klose (doko)]

now fixed in trusty-proposed and vivid in 4.8, and in vivid in 4.9