Dump of assembler code for function test:
=> 0x0000000000400530 <+0>: movzbl %dil,%eax
0x0000000000400534 <+4>: neg %eax
0x0000000000400536 <+6>: retq
So it looks like it is doing all 8 bit math and since 255 in 8 bits is -1, optimizes it to a negation. Definitely looks like an optimizer bug. It should be promoting them to at least unsigned shorts. Even (unsigned short)x * (unsigned short)255UL does not change it.
Oh wow, this is the disassembly I get:
Dump of assembler code for function test: 400534 <+4>: neg %eax 400536 <+6>: retq
=> 0x0000000000400530 <+0>: movzbl %dil,%eax
0x0000000000
0x0000000000
So it looks like it is doing all 8 bit math and since 255 in 8 bits is -1, optimizes it to a negation. Definitely looks like an optimizer bug. It should be promoting them to at least unsigned shorts. Even (unsigned short)x * (unsigned short)255UL does not change it.