Server hangs on updating an XtraDB table after FLUSH TABLES WITH READ LOCK AND DISABLE CHECKPOINT

[Elena Stepanova]

The following sequence of commands

FLUSH TABLES WITH READ LOCK AND DISABLE CHECKPOINT;
UNLOCK TABLES;
CREATE TABLE t1 ( m MEDIUMTEXT ) ENGINE=InnoDB;
INSERT INTO t1 VALUES ( REPEAT('i',1048576) );

makes server to hang, seemingly forever, when it's run with XtraDB (but not with InnoDB plugin).
The above 4 statements are the full test case.

bzr version-info
revision-id: <email address hidden>
date: 2012-05-04 17:22:40 +0200
build-date: 2012-05-05 00:36:39 +0300
revno: 3397

Stack traces of the threads that are doing something (although maybe they are irrelevant to the problem):

Thread 5 (Thread 0x7f8b23b3e710 (LWP 30444)):
#0 0x00007f8b34faa39c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000115fcf2 in safe_cond_wait (cond=0x3112240, mp=0x3112190,
    file=0x13a0838 "storage/xtradb/os/os0sync.c", line=207)
    at mysys/thr_mutex.c:496
#2 0x0000000001067326 in os_cond_wait (cond=0x3112240, mutex=0x3112190)
    at storage/xtradb/os/os0sync.c:207
#3 0x00000000010680ff in os_event_wait_low (event=0x3112190, reset_sig_count=1)
    at storage/xtradb/os/os0sync.c:609
#4 0x0000000000e8659b in sync_array_wait_event (arr=0x260aff0, index=1)
    at storage/xtradb/sync/sync0arr.c:458
#5 0x0000000000e88d95 in rw_lock_s_lock_spin (lock=0x7f8b28d84d88, pass=2,
    file_name=0x1375750 "storage/xtradb/buf/buf0flu.c", line=1335)
    at storage/xtradb/sync/sync0rw.c:453
#6 0x0000000000f45444 in rw_lock_s_lock_func (lock=0x7f8b28d84d88, pass=2,
    file_name=0x1375750 "storage/xtradb/buf/buf0flu.c", line=1335)
    at storage/xtradb/include/sync0rw.ic:419
#7 0x0000000000f45700 in pfs_rw_lock_s_lock_func (lock=0x7f8b28d84d88, pass=2,
    file_name=0x1375750 "storage/xtradb/buf/buf0flu.c", line=1335)
    at storage/xtradb/include/sync0rw.ic:783
#8 0x0000000000f4ce6a in buf_flush_page (buf_pool=0x30cb6b8, bpage=0x7f8b28d84c10, flush_type=BUF_FLUSH_LIST)
    at storage/xtradb/buf/buf0flu.c:1334
#9 0x0000000000f4d583 in buf_flush_try_neighbors (space=0, offset=242, flush_type=BUF_FLUSH_LIST, n_flushed=7,
    n_to_flush=200) at storage/xtradb/buf/buf0flu.c:1519
#10 0x0000000000f4da83 in buf_flush_page_and_try_neighbors (bpage=0x7f8b28d84c10, flush_type=BUF_FLUSH_LIST,
    n_to_flush=200, count=0x7f8b23b3d620) at storage/xtradb/buf/buf0flu.c:1606
#11 0x0000000000f4e16c in buf_flush_flush_list_batch (buf_pool=0x30cb6b8, min_n=200,
    lsn_limit=18446744073709551615) at storage/xtradb/buf/buf0flu.c:1739
#12 0x0000000000f4e341 in buf_flush_batch (buf_pool=0x30cb6b8, flush_type=BUF_FLUSH_LIST, min_n=200,
    lsn_limit=18446744073709551615) at storage/xtradb/buf/buf0flu.c:1829
#13 0x0000000000f4ed9d in buf_flush_list (min_n=200, lsn_limit=18446744073709551615)
    at storage/xtradb/buf/buf0flu.c:2071
#14 0x0000000000e7c1f7 in srv_master_thread (arg=0x0)
    at storage/xtradb/srv/srv0srv.c:3729
#15 0x00007f8b34fa5a4f in start_thread () from /lib64/libpthread.so.0
#16 0x00007f8b33d3a82d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f8b22af4710 (LWP 30454)):
#0 0x00007f8b34faa39c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000000000115fcf2 in safe_cond_wait (cond=0x3258750, mp=0x32586a0,
    file=0x13a0838 "storage/xtradb/os/os0sync.c", line=207)
    at mysys/thr_mutex.c:496
#2 0x0000000001067326 in os_cond_wait (cond=0x3258750, mutex=0x32586a0)
    at storage/xtradb/os/os0sync.c:207
#3 0x00000000010680ff in os_event_wait_low (event=0x32586a0, reset_sig_count=3)
    at storage/xtradb/os/os0sync.c:609
#4 0x0000000000e8659b in sync_array_wait_event (arr=0x260aff0, index=0)
    at storage/xtradb/sync/sync0arr.c:458
#5 0x0000000000e88d95 in rw_lock_s_lock_spin (lock=0x3155b88, pass=0,
    file_name=0x1398ce8 "storage/xtradb/log/log0log.c", line=2076)
    at storage/xtradb/sync/sync0rw.c:453
#6 0x0000000001036836 in rw_lock_s_lock_func (lock=0x3155b88, pass=0,
    file_name=0x1398ce8 "storage/xtradb/log/log0log.c", line=2076)
    at storage/xtradb/include/sync0rw.ic:419
#7 0x0000000001037006 in pfs_rw_lock_s_lock_func (lock=0x3155b88, pass=0,
    file_name=0x1398ce8 "storage/xtradb/log/log0log.c", line=2076)
    at storage/xtradb/include/sync0rw.ic:783
#8 0x000000000103d556 in log_checkpoint (sync=1, write_always=1, safe_to_ignore=0)
    at storage/xtradb/log/log0log.c:2076
#9 0x00000000010382d0 in log_fsp_current_free_limit_set_and_checkpoint (limit=6)
    at storage/xtradb/log/log0log.c:192
#10 0x0000000000fd3acf in fsp_fill_free_list (init_space=0, space=0, header=0x7f8b29370026 "",
    mtr=0x7f8b22af1da0) at storage/xtradb/fsp/fsp0fsp.c:1340
#11 0x0000000000fd41f0 in fsp_alloc_free_extent (space=0, zip_size=0, hint=0, mtr=0x7f8b22af1da0)
    at storage/xtradb/fsp/fsp0fsp.c:1460
#12 0x0000000000fd44da in fsp_alloc_free_page (space=0, zip_size=0, hint=0, mtr=0x7f8b22af1da0)
    at storage/xtradb/fsp/fsp0fsp.c:1522
#13 0x0000000000fd936b in fseg_alloc_free_page_low (space=0, zip_size=0, seg_inode=0x7f8b293533f2 "", hint=0,
    direction=113 'q', mtr=0x7f8b22af1da0) at storage/xtradb/fsp/fsp0fsp.c:2662
#14 0x0000000000fd9c53 in fseg_alloc_free_page_general (seg_header=0x7f8b2934c04a "", hint=320,
    direction=113 'q', has_done_reservation=1, mtr=0x7f8b22af1da0)
    at storage/xtradb/fsp/fsp0fsp.c:2822
#15 0x0000000000ee3e81 in btr_page_alloc (index=0x32e6038, hint_page_no=320, file_direction=113 'q', level=0,
    mtr=0x7f8b22af1da0) at storage/xtradb/btr/btr0btr.c:950
#16 0x0000000000f0f537 in btr_store_big_rec_extern_fields_func (index=0x32e6038, rec_block=0x7f8b28d849c0,
    rec=0x7f8b2934c080 "", offsets=0x2604688, local_mtr=0x7f8b22af2390, update_in_place=0,
    big_rec_vec=0x3730788) at storage/xtradb/btr/btr0cur.c:4386
#17 0x00000000010cbf66 in row_ins_index_entry_low (mode=2, index=0x32e6038, entry=0x32c9ac8, n_ext=0,
    thr=0x3366638) at storage/xtradb/row/row0ins.c:2157
#18 0x00000000010cc14c in row_ins_index_entry (index=0x32e6038, entry=0x32c9ac8, n_ext=0, foreign=1,
    thr=0x3366638) at storage/xtradb/row/row0ins.c:2206
#19 0x00000000010cc740 in row_ins_index_entry_step (node=0x3366410, thr=0x3366638)
    at storage/xtradb/row/row0ins.c:2291
#20 0x00000000010ccc8b in row_ins (node=0x3366410, thr=0x3366638)
    at storage/xtradb/row/row0ins.c:2423
#21 0x00000000010cd1bb in row_ins_step (thr=0x3366638)
    at storage/xtradb/row/row0ins.c:2540
#22 0x0000000000e2ea49 in row_insert_for_mysql (mysql_rec=0x33e5bc8 "\376", prebuilt=0x32c2178)
    at storage/xtradb/row/row0mysql.c:1217
#23 0x0000000000de751c in ha_innobase::write_row (this=0x33c74d8, record=0x33e5bc8 "\376")
    at storage/xtradb/handler/ha_innodb.cc:5820
#24 0x0000000000967bc3 in handler::ha_write_row (this=0x33c74d8, buf=0x33e5bc8 "\376")
    at sql/handler.cc:5064
#25 0x00000000006776af in write_record (thd=0x3290da0, table=0x32dfdb0, info=0x7f8b22af2da0)
    at sql/sql_insert.cc:1800
#26 0x0000000000673eb3 in mysql_insert (thd=0x3290da0, table_list=0x3393370, fields=..., values_list=...,
    update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false)
    at sql/sql_insert.cc:965
#27 0x00000000006a42ab in mysql_execute_command (thd=0x3290da0) at sql/sql_parse.cc:2917
#28 0x00000000006b1305 in mysql_parse (thd=0x3290da0,
    rawbuf=0x3393268 "INSERT INTO t1 VALUES ( REPEAT('i',1048576) )", length=45, parser_state=0x7f8b22af3610)
    at sql/sql_parse.cc:5731
#29 0x000000000069cd8e in dispatch_command (command=COM_QUERY, thd=0x3290da0,
    packet=0x338ba51 "INSERT INTO t1 VALUES ( REPEAT('i',1048576) )", packet_length=45)
    at sql/sql_parse.cc:1055
#30 0x000000000069b8be in do_command (thd=0x3290da0) at sql/sql_parse.cc:794
#31 0x000000000083309d in do_handle_one_connection (thd_arg=0x3290da0)
    at sql/sql_connect.cc:1253
#32 0x00000000008328aa in handle_one_connection (arg=0x3290da0)
    at sql/sql_connect.cc:1168
#33 0x0000000000dd126e in pfs_spawn_thread (arg=0x34e29f0)
    at storage/perfschema/pfs.cc:1015
#34 0x00007f8b34fa5a4f in start_thread () from /lib64/libpthread.so.0
#35 0x00007f8b33d3a82d in clone () from /lib64/libc.so.6

[Elena Stepanova (elenst)]

Also filed in JIRA as https://mariadb.atlassian.net/browse/MDEV-254

[Elena Stepanova (elenst)]

Fix released with 5.5.24