Comment 4 for bug 892951

filesort.cc::make_sortkey converts both "hello" and "HELLO" to sort keys that start with "HELLO". These keys compare identically up to the last part that is the file position (set in make_sortkey around line 992). The fileposition for MEMORY is the memory address and it is stored by direct copying, that is, lowest-order byte first.

Thus the lowest address bytes become more significant in the key comparison, resulting in non-determinant behaviour.

For example, a test run where "hello" sorts before "HELLO":

heap_position = 0x323cbb2 key: hello, filepos = 0xB2 0xCB 0x23 0x3 0x0 0x0 0x0 0x0
heap_position = 0x323ccc6 key: HELLO, filepos = 0xC6 0xCC 0x23 0x3 0x0 0x0 0x0 0x0

A test run where "hello" sorts after "HELLO":

heap_position = 0x1c4f3f2 key: hello, filepos = 0xF2 0xF3 0xC4 0x1 0x0 0x0 0x0 0x0
heap_position = 0x1c4f506 key: HELLO, filepos = 0x6 0xF5 0xC4 0x1 0x0 0x0 0x0 0x0

A fix will require taking into account host endianness and copying the file position with MSBs first.