Non-deterministic percona_heap_blob test failure

[Laurynas Biveinis]

http://jenkins.percona.com/view/Percona%20Server%205.5/job/percona-server-5.5-trunk/74/BUILD_TYPE=release,Host=centos6-32/testReport/junit/%28root%29/main/percona_heap_blob/:

Comment:

Logfile:
CURRENT_TEST: main.percona_heap_blob
--- /home/jenkins/workspace/percona-server-5.5-trunk/BUILD_TYPE/release/Host/centos6-32/Percona-Server-5.5.17-rel21.0/mysql-test/r/percona_heap_blob.result 2011-11-17 15:40:06.314178308 +0300
+++ /home/jenkins/workspace/percona-server-5.5-trunk/BUILD_TYPE/release/Host/centos6-32/Percona-Server-5.5.17-rel21.0/mysql-test/r/percona_heap_blob.reject 2011-11-17 17:57:08.443076901 +0300
@@ -153,8 +153,8 @@

 1
 a
-hello
 HELLO
+hello
 HELLO MY
 select c from t1 order by c;
 c
@@ -163,8 +163,8 @@

 a
-hello
 HELLO
+hello
 HELLO MY
 select b from t1 order by b;
 b

mysqltest: Result content mismatch

[Laurynas Biveinis (laurynas-biveinis)]

I have reviewed the code and did not find any apparent nondeterminism sources. Leaving this with a CI collation to see if it comes up again in the future. Also, will consider additional Valgrind annotations if applicable.

[Alexey Kopytov (akopytov)]

A recent occurrence of this failure: http://jenkins.percona.com/job/percona-server-5.5-param/287/BUILD_TYPE=release,Host=ubuntu-natty-64bit/testReport/junit/%28root%29/main/percona_heap_blob/

[Laurynas Biveinis (laurynas-biveinis)]

Fortunately it is reproducable quite consistently on my machine with --manual-gdb.

A minimal testcase:

--disable_warnings
DROP TABLE IF EXISTS t1;
--enable_warnings

create table t1 (t text,c char(10),b blob, d varbinary(10)) ENGINE=MEMORY;
insert into t1 values (NULL,NULL,NULL,NULL);
insert into t1 values ("","","","");
insert into t1 values ("hello","hello","hello","hello");
insert into t1 values ("HELLO","HELLO","HELLO","HELLO");

select t from t1 order by t;
select c from t1 order by c;

DROP TABLE t1;

[Laurynas Biveinis (laurynas-biveinis)]

filesort.cc::make_sortkey converts both "hello" and "HELLO" to sort keys that start with "HELLO". These keys compare identically up to the last part that is the file position (set in make_sortkey around line 992). The fileposition for MEMORY is the memory address and it is stored by direct copying, that is, lowest-order byte first.

Thus the lowest address bytes become more significant in the key comparison, resulting in non-determinant behaviour.

For example, a test run where "hello" sorts before "HELLO":

heap_position = 0x323cbb2 key: hello, filepos = 0xB2 0xCB 0x23 0x3 0x0 0x0 0x0 0x0
heap_position = 0x323ccc6 key: HELLO, filepos = 0xC6 0xCC 0x23 0x3 0x0 0x0 0x0 0x0

A test run where "hello" sorts after "HELLO":

heap_position = 0x1c4f3f2 key: hello, filepos = 0xF2 0xF3 0xC4 0x1 0x0 0x0 0x0 0x0
heap_position = 0x1c4f506 key: HELLO, filepos = 0x6 0xF5 0xC4 0x1 0x0 0x0 0x0 0x0

A fix will require taking into account host endianness and copying the file position with MSBs first.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1225