DBT2 database load fails

[Seppo Jaakola]

Loading of 30 warehouse DBT2 test database fails in 3 node cluster.
The command to load was:

 mysql -uroot -prootpass -habyssinian dbt2 < dbt2_30wh.sql

Where dbt2_30wh.sql is mysqldump from a 30 warehouse DBT2 database

As a result, the first node hangs in alter table command:

| 6 | root | 10.0.0.101:58402 | dbt2 | Query | 6353 | freeing items | /*!40000 ALTER TABLE `order_line` ENABLE KEYS */ ||

[Seppo Jaakola (seppo-jaakola)]

Appears that the master node crashed:

100125 22:39:08 [ERROR] WSREP: failed to allocate write set items 1483033697-0 for 780
*** glibc detected *** /home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld: corrupted double-linked list: 0x00007f1f2153d5a0 ***
======= Backtrace: =========
/lib/libc.so.6[0x7f1f4a65acb8]
/lib/libc.so.6[0x7f1f4a65e333]
/lib/libc.so.6(__libc_malloc+0x98)[0x7f1f4a65f828]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/galera/lib/libwsdb.so.2[0x7f1f29716282]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/galera/lib/libwsdb.so.2(wsdb_get_write_set+0x1e5)[0x7f1f297169c5]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/galera/lib/libmmgalera.so[0x7f1f299289a2]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld[0x741d18]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld(_Z19ha_commit_one_phaseP3THDb+0x7c)[0x6a2a76]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld(_Z15ha_commit_transP3THDb+0x181)[0x6a2cdb]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld(_Z25ha_autocommit_or_rollbackP3THDi+0x22)[0x6a2f44]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x28e)[0x5c4c5c]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld(_Z10do_commandP3THD+0x1fb)[0x5c6ae1]
/home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld(handle_one_connection+0x5a9)[0x5b3873]
/lib/libpthread.so.0[0x7f1f4b2373ba]
/lib/libc.so.6(clone+0x6d)[0x7f1f4a6c8fcd]
======= Memory map: ========
00400000-00ae6000 r-xp 00000000 fc:00 3621239 /home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld
00ce6000-00d45000 rw-p 006e6000 fc:00 3621239 /home/galera/mysql-5.1.42-galera-0.7.2-x86_64/mysql/libexec/mysqld
00d45000-00d5d000 rw-p 00d45000 00:00 0
00e0a000-02b6e000 rw-p 00e0a000 00:00 0 [heap]
7f1e98000000-7f1e9c000000 rw-p 7f1e98000000 00:00 0
7f1ea0000000-7f1ea3ffe000 rw-p 7f1ea0000000 00:00 0
7f1ea3ffe000-7f1ea4000000 ---p 7f1ea3ffe000 00:00 0
7f1eec000000-7f1eee920000 rw-p 7f1eec000000 00:00 0
7f1eee920000-7f1ef0000000 ---p 7f1eee920000 00:00 0
7f1f05f6f000-7f1f07778000 rw-p 7f1f05f6f000 00:00 0
7f1f18000000-7f1f1c000000 rw-p 7f1f18000000 00:00 0
7f1f1f223000-7f1f1f455000 rw-p 7f1f1f223000 00:00 0
7f1f20000000-7f1f24000000 rw-p 7f1f20000000 00:00 0
7f1f25064000-7f1f27064000 rw-p 7f1f25064000 00:00 0
7f1f27064000-7f1f27078000 r-xp 00000000 fc:00 2187299 /lib/libresolv-2.9.so
7f1f27078000-7f1f27278000 ---p 00014000 fc:00 2187299 /lib/libresolv-2.9.so
7f1f27278000-7f1f27279000 r--p 00014000 fc:00 2187299 /lib/libresolv-2.9.so
7f1f27279000-7f1f2727a000 rw-p 00015000 fc:00 2187299 /lib/libresolv-2.9.so
7f1f2727a000-7f1f2727c000 rw-p 7f1f2727a000 00:00 0
7f1f2727c000-7f1f27281000 r-xp 00000000 fc:00 2187292 /lib/libnss_dns-2.9.so
7f1f27281000-7f1f27480000 ---p 00005000 fc:00 2187292 /lib/libnss_dns-2.9.so
7f1f27480000-7f1f27481000 r--p 00004000 fc:00 2187292 /...

[Seppo Jaakola (seppo-jaakola)]

I had to unpack gdb from my toolbox to sort this out. And what turned out was:

1. The DBT2 dump file contains LOCK TABLE ... UNLOCK TABLE around all inserts for each table
  (mysqldump can be configured to use LOCKs or not, but this dump now happened to use locks)

+

2. wsrep config has by default setting to convert locking sessions to transactions
  (wsrep_convert_LOCK_to_trx variable)

=> this resulted in jumbo transactions
      and wsdb failed to malloc space for the write set

Lesson to learn is:
* not a good idea to convert locks to trx by default
* wsdb should report out of mem situation in a better way

The dump file loads fine, if my.cnf has: wsrep_convert_LOCK_to_trx=0