Segfault with incremental backup for innodb_page_size=8K and with encrypted, compressed, with json and generated column table

[Shahriyar Rzayev]

Hi,
I have started PS 5.7 with:

--innodb_buffer_pool_size=1G --innodb_log_file_size=1G --innodb_page_size=8K --early-plugin-load=keyring_file.so --keyring_file_data=/home/shahriyar.rzaev/XB_TEST/server_dir/PS211117-percona-server-5.7.19-17-linux-x86_64-debug/mysql-keyring/keyring --log-bin=mysql-bin --log-slave-updates --server-id=2 --gtid-mode=ON --enforce-gtid-consistency --binlog-format=row

Run sysbench:
sysbench /usr/share/sysbench/oltp_insert.lua --table-size=1000 --tables=30 --mysql-db=sysbench_test_db --mysql-user=root --threads=100 --db-driver=mysql --mysql-socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS211117-percona-server-5.7.19-17-linux-x86_64-debug/socket.sock prepare

And did some sort of things with several of these tables:

            for i in range(1, 5):
                sql_encrypt = "alter table sysbench_test_db.sbtest{} encryption='Y'".format(i)
                RunBenchmark.run_sql_statement(basedir=self.basedir, sql_statement=sql_encrypt)
                sql_compress = "alter table sysbench_test_db.sbtest{} compression='lz4'".format(i)
                RunBenchmark.run_sql_statement(basedir=self.basedir, sql_statement=sql_compress)
                sql_optimize = "optimize table sysbench_test_db.sbtest{}".format(i)
                RunBenchmark.run_sql_statement(basedir=self.basedir, sql_statement=sql_optimize)
                sql_virtual_column = "alter table sysbench_test_db.sbtest{} add column json_test_v json generated always as (json_array(k,c,pad)) virtual".format(i)
                RunBenchmark.run_sql_statement(basedir=self.basedir, sql_statement=sql_virtual_column)
                sql_stored_column = "alter table sysbench_test_db.sbtest{} add column json_test_s json generated always as (json_array(k,c,pad)) stored".format(i)
                RunBenchmark.run_sql_statement(basedir=self.basedir, sql_statement=sql_stored_column)
                sql_create_json_column = "alter table sysbench_test_db.sbtest{} add column json_test_index varchar(255) generated always as (json_array(k,c,pad)) stored".format(i)
                RunBenchmark.run_sql_statement(basedir=self.basedir, sql_statement=sql_create_json_column)
                sql_alter_add_index = "alter table sysbench_test_db.sbtest{} add index(json_test_index)".format(i)
                RunBenchmark.run_sql_statement(basedir=self.basedir, sql_statement=sql_alter_add_index)
            for i in range(5, 10):
                sql_compress = "alter table sysbench_test_db.sbtest{} compression='zlib'".format(i)
                RunBenchmark.run_sql_statement(basedir=self.basedir, sql_statement=sql_compress)

After that took full backup:

/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup --defaults-file= --user=root --password='' --target-dir=/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/cycle2/full/2017-11-22_21-18-43 --backup --socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS211117-percona-server-5.7.19-17-linux-x86_64-debug/socket.sock --compress=quicklz --compress-chunk-size=65536 --compress-threads=4 --encrypt=AES256 --encrypt-key=VVTBwgM4UhwkTTV98fhuj+D1zyWoA89K --encrypt-threads=4 --encrypt-chunk-size=65536 --slave-info --no-version-check --core-file --parallel=1 --throttle=40 --keyring-file-data=/home/shahriyar.rzaev/XB_TEST/server_dir/PS211117-percona-server-5.7.19-17-linux-x86_64-debug/mysql-keyring/keyring --stream="xbstream" > /home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/cycle2/full/2017-11-22_21-18-43/full_backup.stream

Run sysbench:

sysbench /usr/share/sysbench/oltp_update_non_index.lua --table-size=1000 --tables=100 --mysql-db=sysbench_test_db --mysql-user=root --threads=100 --db-driver=mysql --mysql-socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS211117-percona-server-5.7.19-17-linux-x86_64-debug/socket.sock run

Taking incremental backup:

2017-11-22 21:19:35 DEBUG The following xbstream command will be executed /home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xbstream -x --parallel=100 --decrypt=AES256 --encrypt-key=VVTBwgM4UhwkTTV98fhuj+D1zyWoA89K --encrypt-threads=4 < /home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/cycle2/full/2017-11-22_21-18-43/full_backup.stream -C /home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/cycle2/full/2017-11-22_21-18-43
2017-11-22 21:19:49 DEBUG OK: XBSTREAM command succeeded.
2017-11-22 21:19:49 WARNING Streaming is enabled!
2017-11-22 21:19:49 DEBUG The following backup command will be executed /home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup --defaults-file= --user=root --password='' --target-dir=/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/cycle2/inc/2017-11-22_21-19-35 --incremental-basedir=/home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/cycle2/full/2017-11-22_21-18-43 --backup --socket=/home/shahriyar.rzaev/XB_TEST/server_dir/PS211117-percona-server-5.7.19-17-linux-x86_64-debug/socket.sock --compress=quicklz --compress-chunk-size=65536 --compress-threads=4 --encrypt=AES256 --encrypt-key=VVTBwgM4UhwkTTV98fhuj+D1zyWoA89K --encrypt-threads=4 --encrypt-chunk-size=65536 --slave-info --no-version-check --core-file --parallel=1 --throttle=40 --keyring-file-data=/home/shahriyar.rzaev/XB_TEST/server_dir/PS211117-percona-server-5.7.19-17-linux-x86_64-debug/mysql-keyring/keyring --stream="xbstream" > /home/shahriyar.rzaev/XB_TEST/backup_dir/ps_5_7_x_2_4/cycle2/inc/2017-11-22_21-19-35/inc_backup.stream
171122 21:19:55 [01] Compressing, encrypting and streaming ./sysbench_test_db/sbtest9.ibd
171122 21:19:55 [01] ...done
171122 21:19:55 [01] Compressing, encrypting and streaming ./sysbench_test_db/sbtest3.ibd
21:19:55 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup(my_print_stacktrace+0x35)[0x17f6fce]
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup(handle_fatal_signal+0x2ab)[0x1165f8f]
/usr/lib64/libpthread.so.0(+0xf130)[0x7ffa2da76130]
/usr/lib64/libc.so.6(+0x149c00)[0x7ffa2b889c00]
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup[0x11581bb]
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup[0x115aa86]
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup(LZ4_decompress_fast+0x6a)[0x115acd2]
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup(_ZN11Compression11deserializeEbPhS0_m+0x20d)[0xf378eb]
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup(_Z23os_file_decompress_pagebPhS_m+0x30)[0xf37ad0]
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup(_Z15xb_fil_cur_readP12xb_fil_cur_t+0x4a7)[0xddb447]
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup[0xdbafe6]
/home/shahriyar.rzaev/XB_TEST/server_dir/target/percona-xtrabackup-2.4.x-debug/bin/xtrabackup[0xdbbefc]
/usr/lib64/libpthread.so.0(+0x7df5)[0x7ffa2da6edf5]
/usr/lib64/libc.so.6(clone+0x6d)[0x7ffa2b8361ad]

For full log please see attachment.

[Shahriyar Rzayev (rzayev-sehriyar)]

(gdb) bt
+bt
#0 0x00007ffff59e1c00 in __memcpy_ssse3_back () from /usr/lib64/libc.so.6
#1 0x00000000011581bb in LZ4_copy8 (dstPtr=0x7fffc8002359, srcPtr=0x7fffc7ffc886) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/extra/lz4/lz4.c:204
#2 0x000000000115aa86 in LZ4_decompress_generic (source=0x7fffc8002986 "\361\004", dest=0x7fffc8000950 "", inputSize=0, outputSize=8154, endOnInput=0, partialDecoding=0, targetOutputSize=0, dict=1, lowPrefix=0x7fffc7ff0950 "", dictStart=0x0, dictSize=65536) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/extra/lz4/lz4.c:1258
#3 0x000000000115acd2 in LZ4_decompress_fast (source=0x7fffc8002986 "\361\004", dest=0x7fffc8000950 "", originalSize=8154) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/extra/lz4/lz4.c:1300
#4 0x0000000000f378eb in Compression::deserialize (dblwr_recover=false, src=0x7fffc8002960 "Z1", <incomplete sequence \357>, dst=0x7fffc8000950 "", dst_len=8192) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/storage/innobase/os/os0file.cc:8555
#5 0x0000000000f37ad0 in os_file_decompress_page (dblwr_recover=false, src=0x7fffc8002960 "Z1", <incomplete sequence \357>, dst=0x7fffc8000950 "", dst_len=8192) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/storage/innobase/os/os0file.cc:8618
#6 0x0000000000ddb447 in xb_fil_cur_read (cursor=0x7fffd2bfd7a0) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/storage/innobase/xtrabackup/src/fil_cur.cc:380
#7 0x0000000000dbafe6 in xtrabackup_copy_datafile (node=0x2b586c0, thread_n=1) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/storage/innobase/xtrabackup/src/xtrabackup.cc:2731
#8 0x0000000000dbbefc in data_copy_thread_func (arg=0x2a93220) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/storage/innobase/xtrabackup/src/xtrabackup.cc:3270
#9 0x00007ffff7bc6df5 in start_thread () from /usr/lib64/libpthread.so.0
#10 0x00007ffff598e1ad in clone () from /usr/lib64/libc.so.6

[Shahriyar Rzayev (rzayev-sehriyar)]

(gdb) bt full
+bt full
#0 0x00007ffff59e1c00 in __memcpy_ssse3_back () from /usr/lib64/libc.so.6
No symbol table info available.
#1 0x00000000011581bb in LZ4_copy8 (dstPtr=0x7fffc8002359, srcPtr=0x7fffc7ffc886) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/extra/lz4/lz4.c:204
No locals.
#2 0x000000000115aa86 in LZ4_decompress_generic (source=0x7fffc8002986 "\361\004", dest=0x7fffc8000950 "", inputSize=0, outputSize=8154, endOnInput=0, partialDecoding=0, targetOutputSize=0, dict=1, lowPrefix=0x7fffc7ff0950 "", dictStart=0x0, dictSize=65536) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/extra/lz4/lz4.c:1258
        token = 157
        length = 17
        match = 0x7fffc7ffc886 ""
        ip = 0x7fffc8002b43 "B\314\372\\Z^\034{0JӰ\306\bE_:\034:\341\024\224z^\212O\316ݶ\337/\001\203B\ni\340f\344\340[Z\024\r\226Uo}\374\372\366A\024a\351\325A\030/^J\026\065o0\231\342>\353\345\217c\255\327\n\321⃟9\231\334ة\215g\340J\262\240$\254\201\"\t!\003\344\330g\373\333\017\230\257J\306bz\271b\237a\336mz]\234a\360\003&\214\222ץ\233\027\063\351\323\030\200\337ն\036Rl\212\001\024B\367\202\004\371N&\303#\204@j(\245\241f\035"
        iend = 0x7fffc8002986 "\361\004"
        op = 0x7fffc8002359 "\323ZB\314\372\\Z\221\034U\252\322\034\306\066j\026\370ޞ[\335N\302,\024\275"
        oend = 0x7fffc800292a ""
        cpy = 0x7fffc800236a "\370ޞ[\335N\302,\024\275"
        oexit = 0x7fffc8000950 ""
        lowLimit = 0x7fffc7fe0950 ""
        dictEnd = 0x10000 <Address 0x10000 out of bounds>
        dec32table = {[0] = 4,
          [1] = 1,
          [2] = 2,
          [3] = 1,
          [4] = 4,
          [5] = 4,
          [6] = 4,
          [7] = 4}
        dec64table = {[0] = 0,
          [1] = 0,
          [2] = 0,
          [3] = 18446744073709551615,
          [4] = 0,
          [5] = 1,
          [6] = 2,
          [7] = 3}
        safeDecode = 0
        checkOffset = 0
#3 0x000000000115acd2 in LZ4_decompress_fast (source=0x7fffc8002986 "\361\004", dest=0x7fffc8000950 "", originalSize=8154) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/extra/lz4/lz4.c:1300
No locals.
#4 0x0000000000f378eb in Compression::deserialize (dblwr_recover=false, src=0x7fffc8002960 "Z1", <incomplete sequence \357>, dst=0x7fffc8000950 "", dst_len=8192) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/storage/innobase/os/os0file.cc:8555
        block = 0x0
        ret = 0
        len = 8154
        header = {
          m_version = 1 '\001',
          m_algorithm = Compression::LZ4,
          m_original_type = 17855,
          m_original_size = 8154,
          m_compressed_size = 450
        }
        ptr = 0x7fffc8002986 "\361\004"
        compression = {
          m_type = Compression::LZ4
        }
#5 0x0000000000f37ad0 in os_file_decompress_page (dblwr_recover=false, src=0x7fffc8002960 "Z1", <incomplete sequence \357>, dst=0x7fffc8000950 "", dst_len=8192) at /mnt/workspace/qa.pxb24.build/BUILD_TYPE/debug/label_exp/centos7-64/storage/innobase/os/os0file.cc:8618
No locals.
#6 0x0000000000ddb447 in xb_fil_cur_read (cursor=0x7fffd2bfd7a0) at /mnt/workspace/qa.pxb24.bui...

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-516