TokuDB Hot Backup Plugin does not release files
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona Server moved to https://jira.percona.com/projects/PS |
Incomplete
|
Undecided
|
Unassigned |
Bug Description
Hello.
I run a backup job for my Percona server with TokuDB engine as a cronjob task:
time mysql -e "SET tokudb_
/var/backup/mysql - is a mount point for EBS volume
When backup job is finished I see that mysqld process does not release files in backup directory:
# tail -n2 /var/log/mysql.log
Toku Hot Backup: Finished: Mon Aug 7 19:12:18 2017
# lsof -p $(pidof mysqld) | grep "backup" | tail -n 10
mysqld 28603 mysql 2016u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2021u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2022u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2023u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2031u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2032u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2033u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2034u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2039u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2055u REG 202,80 112 44302937 /var/backup/
# lsof -p $(pidof mysqld) | grep -c "backup"
852
If I delete these files, descriptors mark as deleted, what is expected:
# rm -rf /var/backup/mysql/*
# lsof -p $(pidof mysqld) | grep "backup" | tail -n 10
mysqld 28603 mysql 2016u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2021u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2022u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2023u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2031u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2032u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2033u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2034u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2039u REG 202,80 112 44302937 /var/backup/
mysqld 28603 mysql 2055u REG 202,80 112 44302937 /var/backup/
Every time I run backup job, the number of open descriptors increases until they begin to rest on the limit and backup job crashes with 'too many open files' error.
Restart of mysqld process helps to clear deleted/opened descriptors, but this is bad solution I think :)
A similar problem I found on the percona forum:
https:/
# cat /etc/issue
Amazon Linux AMI release 2017.03
Kernel \r on an \m
# rpm -qa | grep Percona
Percona-
Percona-
Percona-
Percona-
Hi Anton,
I'm not able to reproduce the issue with single partitioned tokudb table without significant load.
Do you have file descriptors leak with tokudb_backup_dir located on same file system as mysql datadir?
Please provide the output for (during backup): /environ| xargs -0 -n 1|grep LD_PRELOAD
cat /proc/`pgrep -xn mysqld`
pt-summary
pt-mysql-summary