gcc optimization problem related to varargs

[Christian Buchner]

Binary package hint: gcc-4.4

This is reproducable on Ubuntu lucid (development branch) 10.04

Passing a variable argument list to vsnprintf, of which I take the fifth argument and
print the output to the console. Works without compiler optimization, breaks starting
-O1 through -O3.

[code]
#include <stdio.h>
#include <stdarg.h>

void testfunc(char *string, ...)
{
    char buf[256];

    va_list arglist;
    va_start(arglist, string);

    vsnprintf(buf, sizeof(buf), string, arglist);
    puts(buf);
}

int main(int argc, char **argv)
{
    testfunc("TEST=%5$sn", 1, 2, 3, 4, "Hello");
}
[/code]

Compile with
g++ -O0 -o test test.cpp
and
g++ -O3 -o test test.cpp
respectively.

Same behaviour when gcc is invoked instead of g++

Expected output:
TEST=Hello

getting instead (with -O1 or higher)
*** invalid %N$ use detected ***

Likely a compiler optimization issue, less likely an internal problem inside glibc. Possibly also a user error.
If it is a user error, glibc should also complain with -O0, right?

This g++ behaviour prevents our application from working on Ubuntu when compiled in release (optimized) mode.

Christian Buchner

[Colin Watson (cjwatson)]

(I think there's an extra "n" in your first argument to testfunc(), at
least when compared to your expected output.)

Thanks for your report. What's happening here is that, when you pass
five arguments to vsnprintf but only consume the fifth, vsnprintf has no
idea what the types of the other four arguments are. Due to the way
variable-length argument lists work, this means that vsnprintf doesn't
know the position of the fifth argument on the stack.

The reason that it doesn't fail with -O0 is that Ubuntu's gcc sets
-D_FORTIFY_SOURCE=2 at -O2 and higher. When _FORTIFY_SOURCE is not set,
all positional arguments are assumed to be int unless there's evidence
to the contrary. Otherwise, any undefined-type positional arguments
produce this error. (I think %N$ is a GNU extension, so it seems within
its rights to behave this way, and it's not an unreasonable security
check since this can at the very least cause surprising behaviour,
particularly in conjunction with %n.)

Thus, a short-term workaround for you is to build with
-U_FORTIFY_SOURCE, but I think you should also change your application
to avoid using undefined-type positional arguments like this. The two
bugs in Ubuntu here are that (as far as I can see) glibc doesn't
document this behaviour, and that gcc doesn't warn about the risky code.

 affects ubuntu/gcc-4.4
 status triaged
 severity wishlist
 affects ubuntu/eglibc
 status triaged
 severity low

[Kees Cook (kees)]

You seem to have a trailing "n" in your format string? i.e. running the attached code results in "TEST=Hellon" for me.

Regardless, this is an invalid format specification. Quoting the printf man page:

       ... There may be no gaps in
       the numbers of arguments specified using '$'; for example, if arguments
       1 and 3 are specified, argument 2 must also be specified somewhere in
       the format string.

In this case the code has skipped 1 through 4. Compile-time cannot check for this since a format string may be dynamically generated. When increasing the optimization level, runtime checks are added.

Why they cannot be skipped, I don't understand. :)

[Kees Cook (kees)]

Please be extremely careful with -U_FORTIFY_SOURCE as disabling this takes away a large number of important protections.

gcc does warn about the behavior when it can detect it at compile time (it cannot with varargs). If you change testfunc to printf:
$ cc -O2 test.c -o test
test.c: In function ‘main’:
test.c:17: warning: format argument 1 unused before used argument 5 in $-style format
test.c:17: warning: format argument 2 unused before used argument 5 in $-style format
test.c:17: warning: format argument 3 unused before used argument 5 in $-style format
test.c:17: warning: format argument 4 unused before used argument 5 in $-style format

[Kees Cook (kees)]

For more details on Ubuntu's proactive security in the compiler, see https://wiki.ubuntu.com/CompilerFlags