xtrabackup binary leaks file descriptors on --backup

Bug #713267 reported by Peter Zaitsev
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
Medium
Alexey Kopytov
1.6
Fix Released
Medium
Alexey Kopytov
2.0
Fix Released
Medium
Alexey Kopytov
2.1
Fix Released
Medium
Alexey Kopytov

Bug Description

Server is configured with innodb_open_files=10000 and open_files_limit=30000

Xtrabackup is backing up 10000+ tables and is crashing with error 24 (out of file descriptors mid way thought)

ulimit -n 30000 before running it fixes the problem.

xtrabackup should handle open_files_limit option and increase number of limits and refuse to run or reduce number of allowed open files if needed. Alternatively we may restrict innodb_open_files setting as xtrabackup does not need many open files for operation.

Related branches

Revision history for this message
Alexey Kopytov (akopytov) wrote :

Peter,

I'm not quite clear about both the bug and the proposed solution.

So xtrabackup crashed because the OS did not allow it to proceed due to open files limit. Note that during backup xtrabackup does not use more that (2 * N + 1) descriptors where N is the number of --parallel threads. The remaining descriptors could be used by the server or some other activity in the system.

I don't see how that can be fixed by handling server settings. It's about the OS limit, rather than any server/InnoDB limit. It's also not clear how cases like that can be detected in advance so xtrabackup would refuse to run.

Changed in percona-xtrabackup:
status: New → Incomplete
Revision history for this message
Peter Zaitsev (pz-percona) wrote :

Alexey,

I see the bug as follows - as Xtrabackup and MySQL Server operate using same my.cnf should work
in both configuration

if I have innodb_open_files_limit=30000 (or similar high number) in my.cnf and open_files_limit=50000

when we'll have as follows

- MySQL Server works fine as it increases number of files allowed based on open_files_limit setting

- Xtrabackup does not work with large number of tables as it does not respect open_files_limit option as I understand.

The solutions could be either:

- support open_files_limit as MySQL server does

- more simple one, do not allow innodb_open_files_limit to be set to more than 512 or so (it should be enough)

- close the table once it is copied closing file descriptor indpendent of open_files_limit setting.

Changed in percona-xtrabackup:
status: Incomplete → Triaged
Revision history for this message
Alexey Kopytov (akopytov) wrote :

Peter,

xtrabackup does close .ibd after copying them. So at any moment it only has up to 3 files open (with no --parallel): source, destination and the log file. It also doesn't use tablespace management functions from InnoDB (where innodb_open_files limit is checked), but instead opens and closes files using the lower-level file management functions.

So what I think actually happened is that the number of files open by the server + 3 (number of files open by xtrabackup) exceeded "ulimit -n" at some point. But that cannot be avoided by handling server/InnoDB file limits. Raising ulimit -n, however, should help, and it did.

Revision history for this message
Peter Zaitsev (pz-percona) wrote : Re: [Bug 713267] Re: Xtrabackup fails if there are many open files allowed

Alexey,

OK lets read the bug when differently.

if innodb_open_files_limit set to 20000 and some 20000 of innodb tables
are backed up the innodb failes with ulimit -n = 1024

which should not happen per your description. Might be it only was the case
in old
version. can you or Valentine check it just to be sure ?

--
Peter Zaitsev, CEO, Percona Inc.
Tel: +1 888 401 3401 ext 501 Skype: peter_zaitsev
24/7 Emergency Line +1 888 401 3401 ext 911

Fantastic value MySQL Support from Percona
http://www.percona.com/mysql-support/

Revision history for this message
Alexey Kopytov (akopytov) wrote :

Peter,

On 27.06.11 21:14, Peter Zaitsev wrote:
> Alexey,
>
> OK lets read the bug when differently.
>
> if innodb_open_files_limit set to 20000 and some 20000 of innodb tables
> are backed up the innodb failes with ulimit -n = 1024
>
> which should not happen per your description. Might be it only was the case
> in old
> version. can you or Valentine check it just to be sure ?
>
>

I have constructed a simple test case and discovered that xtrabackup
indeed leaks file descriptors.

That is, xtrabackup calls fil_space_get_zip_size() with the only purpose
to determine compressed page size for a tablespace. Which in turns calls
fil_node_open_file() eventually than opens the file. But xtrabackup
never calls fil_node_close_file().

So all .ibd files, once opened before they are copied, stay open until
backup completes for no good reason.

Setting the bug to confirmed.

Changed in percona-xtrabackup:
status: Triaged → Confirmed
Stewart Smith (stewart)
Changed in percona-xtrabackup:
importance: Undecided → Medium
status: Confirmed → Triaged
summary: - Xtrabackup fails if there are many open files allowed
+ xtrabackup binary leaks file descriptors on --backup
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-558

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.