Comment 4 for bug 892951 converts both "hello" and "HELLO" to sort keys that start with "HELLO". These keys compare identically up to the last part that is the file position (set in make_sortkey around line 992). The fileposition for MEMORY is the memory address and it is stored by direct copying, that is, lowest-order byte first.

Thus the lowest address bytes become more significant in the key comparison, resulting in non-determinant behaviour.

For example, a test run where "hello" sorts before "HELLO":

heap_position = 0x323cbb2 key: hello, filepos = 0xB2 0xCB 0x23 0x3 0x0 0x0 0x0 0x0
heap_position = 0x323ccc6 key: HELLO, filepos = 0xC6 0xCC 0x23 0x3 0x0 0x0 0x0 0x0

A test run where "hello" sorts after "HELLO":

heap_position = 0x1c4f3f2 key: hello, filepos = 0xF2 0xF3 0xC4 0x1 0x0 0x0 0x0 0x0
heap_position = 0x1c4f506 key: HELLO, filepos = 0x6 0xF5 0xC4 0x1 0x0 0x0 0x0 0x0

A fix will require taking into account host endianness and copying the file position with MSBs first.