Comment 26 for bug 685352

doko asked me to look into this. This is definitely a compiler bug. The minimal test case I get is:

unsigned short test (unsigned char val) __attribute__ ((noinline));

unsigned short
test (unsigned char val)
{
  return val * 255;
}

int
main(int argc, char**argv)
{
  printf ("test(val=40) = %x\n", test(0x40));
  return 0;
}

When built without optimization, we get (correctly):
test(val=40) = 3fc0

When built with optimization, we get instead:
test(val=40) = ffc0

There is nothing undefined in the source code (unsigned arithmetic has defined overflow behaviour in C, and in any case, there actually isn't any overflow in this particular example).

The bug is that GCC realizes that for 8-bit integers, multiplying by 255 is actually the same as just taking the negative value (which is true as far as it goes, if you only use 8 bits of the results) -- and then erroneously does the same optimization when we actually need 16 bits of the result.

This bug seems to have been introduced sometime in the 4.5 cycle, and is actually still present on current mainline ...