mysqldump takes 5x as long with 5.7.20-19

[William Taylor]

Originally posted this on the Percona forums with no response yet. This is probably a better place to post though.

After we upgraded to 5.7.20-19 from 5.7.19, mysql backups for this table jumped from 42 mins to 5hrs.
The backup time for the innodb table doubled but the tokudb table took more than 5 times longer.
I saw there was a memory leak fixed in mysqldump but unless there were other changes there I
don't see that being the issue.

Any ideas?

# Time: 2018-01-04T12:22:36.800014Z
# User@Host: root[root] @ localhost [] Id: 148138
# Schema: cdr Last_errno: 0 Killed: 0
# Query_time: 2568.367009 Lock_time: 0.000000 Rows_sent: 523136203 Rows_examined: 523136203 Rows_affected: 0
# Bytes_sent: 188647575431
SET timestamp=1515068556;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `cdr_main`;

# Time: 2018-01-04T12:22:56.091752Z
# User@Host: root[root] @ localhost [] Id: 148138
# Schema: cdr Last_errno: 0 Killed: 0
# Query_time: 19.015636 Lock_time: 0.000000 Rows_sent: 11859695 Rows_examined: 11859695 Rows_affected: 0
# Bytes_sent: 239650071
SET timestamp=1515068576;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `line_item_ratings`;

After upgrade to 5.7.20-19

# Time: 2018-01-10T20:18:38.510011Z
# User@Host: root[root] @ localhost [] Id: 29239
# Schema: cdr Last_errno: 1681 Killed: 0
# Query_time: 18269.061056 Lock_time: 0.000000 Rows_sent: 525785773 Rows_examined: 525785773 Rows_affected: 0
# Bytes_sent: 189596705933
SET timestamp=1515615518;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `cdr_main`;

# Time: 2018-01-10T20:19:24.966083Z
# User@Host: root[root] @ localhost [] Id: 29239
# Schema: cdr Last_errno: 1681 Killed: 0
# Query_time: 44.798296 Lock_time: 0.000000 Rows_sent: 11859695 Rows_examined: 11859695 Rows_affected: 0
# Bytes_sent: 239650071
SET timestamp=1515615564;
SELECT /*!40001 SQL_NO_CACHE */ * FROM `line_item_ratings`;

SHOW TABLE STATUS:
Name: cdr_main
Engine: TokuDB
Version: 10
Row_format: tokudb_zlib
Rows: 526232788
Avg_row_length: 323
Data_length: 169998403381
Max_data_length: 9223372036854775807
Index_length: 35156877283
Data_free: 18446743920195218452
Auto_increment: 829412264
Create_time: 2016-07-11 11:16:33
Update_time: 2018-01-10 17:08:38
Check_time: NULL
Collation: latin1_swedish_ci
Checksum: NULL
Create_options:
Comment:

Name: line_item_ratings
Engine: InnoDB
Version: 10
Row_format: Dynamic
Rows: 6108827
Avg_row_length: 35
Data_length: 214663168
Max_data_length: 0
Index_length: 261586944
Data_free: 5242880
Auto_increment: NULL
Create_time: 2016-08-24 11:34:06
Update_time: NULL
Check_time: NULL
Collation: latin1_swedish_ci
Checksum: NULL
Create_options:
Comment:

[Roel Van de Paar (roel11)]

@Shako, can you please test this as discussed? Thank you

[Roel Van de Paar (roel11)]

https://www.percona.com/forums/questions-discussions/mysql-and-percona-server/50493-mysqldump-backup-times-greatly-increased-with-5-7-20-19

[Roel Van de Paar (roel11)]

Discussion in the thread.

[Roel Van de Paar (roel11)]

Tests do not show the issue. If we receive further OS/settings info, we can test again.

[William Taylor (williamt-sonic)]

It looks like its not the new mysql version.
It appears to be this kernel "kernel-3.10.0-693.11.6.el7.x86_64"
rolling back to "kernel-3.10.0-693.11.1.el7.x86_64" seems to have resolved the problems.
This is on CentOS Linux release 7.4.1708 (Core)

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3772