pt-table-checksum: Deep recursion on subroutine "SchemaIterator::_iterate_dbh"

Reported by Steven Barre on 2013-02-28
14
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Percona Toolkit
Status tracked in 2.2
2.1
High
Daniel Nichter
2.2
High
Daniel Nichter

Bug Description

Seems related to https://bugs.launchpad.net/percona-toolkit/+bug/1003323 but I wanted to post a new bug since that one is quite old. Feel free to close as dupe

# pt-table-checksum --version
pt-table-checksum 2.1.9

# PTDEBUG=1 pt-table-checksum --tables nwmls.properties_tmp --chunk-size-limit 4 h=localhost,u=XXXX,p=XXXX
<snip output>
# SchemaIterator:6888 7610 Next database: ftmyers_far
# SchemaIterator:6894 7610 SHOW /*!50002 FULL*/ TABLES FROM `ftmyers_far`
# SchemaIterator:7014 7610 Table city_codes is not in --tables list, ignoring
# SchemaIterator:6905 7610 Found 0 tables in database ftmyers_far
# SchemaIterator:6933 7610 No more tables in database ftmyers_far
# SchemaIterator:6872 7610 Getting next schema object from dbh DBI::db=HASH(0xe5d3cd0)
# SchemaIterator:6888 7610 Next database: fwamls
# SchemaIterator:6894 7610 SHOW /*!50002 FULL*/ TABLES FROM `fwamls`
# SchemaIterator:7014 7610 Table sold_property_mult is not in --tables list, ignoring
# SchemaIterator:7014 7610 Table sold_property_parr is not in --tables list, ignoring
# SchemaIterator:7014 7610 Table sold_property_resi is not in --tables list, ignoring
# SchemaIterator:7014 7610 Table sold_property_resr is not in --tables list, ignoring
# SchemaIterator:7014 7610 Table virtual_media is not in --tables list, ignoring
# SchemaIterator:6905 7610 Found 0 tables in database fwamls
# SchemaIterator:6933 7610 No more tables in database fwamls
Deep recursion on subroutine "SchemaIterator::_iterate_dbh" at /root/bin/pt-table-checksum line 6937.
# Cxn:3349 7610 Disconnecting dbh DBI::db=HASH(0xe5dec30) undef
# Cxn:3349 7610 Disconnecting dbh DBI::db=HASH(0xe5e8d10) undef
# Cxn:3349 7610 Disconnecting dbh DBI::db=HASH(0xe5d3cd0) undef

Daniel Nichter (daniel-nichter) wrote :

Steven, can you create a test case to reproduce this? I.e., dump some dbs and tables that would cause this to happen on any system? As in bug 1003323, we'll need some way to reproduce this issue because it seems pretty rare and it has never happened in testing.

tags: added: crash pt-table-checksum schema-iter
Changed in percona-toolkit:
status: New → Incomplete
Steven Barre (steven-8) wrote :

It seems to be just having too many databases causes the issue. The attached sql creates 100 databases and adds a table to the first one.

# cat /etc/redhat-release
CentOS release 5.9 (Final)

# perl -v
This is perl, v5.8.8 built for x86_64-linux-thread-multi

# rpm -qa perl*
perl-DBI-1.52-2.el5
perl-5.8.8-38.el5_8
perl-TermReadKey-2.30-4.el5
perl-String-CRC32-1.4-2.fc6
perl-DBD-MySQL-3.0007-2.el5

# mysql -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 26
Server version: 5.5.30-log Distributed by The IUS Community Project

# PTDEBUG=1 ./pt-table-checksum --tables foo0001.bla01 h=localhost,u=root --ask-pass
<snip>
# SchemaIterator:6888 3789 Next database: foo0097
# SchemaIterator:6894 3789 SHOW /*!50002 FULL*/ TABLES FROM `foo0097`
# SchemaIterator:6905 3789 Found 0 tables in database foo0097
# SchemaIterator:6933 3789 No more tables in database foo0097
# SchemaIterator:6872 3789 Getting next schema object from dbh DBI::db=HASH(0x134adcc0)
# SchemaIterator:6888 3789 Next database: foo0098
# SchemaIterator:6894 3789 SHOW /*!50002 FULL*/ TABLES FROM `foo0098`
# SchemaIterator:6905 3789 Found 0 tables in database foo0098
# SchemaIterator:6933 3789 No more tables in database foo0098
# SchemaIterator:6872 3789 Getting next schema object from dbh DBI::db=HASH(0x134adcc0)
# SchemaIterator:6888 3789 Next database: foo0099
# SchemaIterator:6894 3789 SHOW /*!50002 FULL*/ TABLES FROM `foo0099`
# SchemaIterator:6905 3789 Found 0 tables in database foo0099
# SchemaIterator:6933 3789 No more tables in database foo0099
Deep recursion on subroutine "SchemaIterator::_iterate_dbh" at ./pt-table-checksum line 6937.
# Cxn:3349 3789 Disconnecting dbh DBI::db=HASH(0x134adcc0) undef

Brian Fraser (fraserbn) on 2013-03-05
Changed in percona-toolkit:
status: Incomplete → Confirmed
Brian Fraser (fraserbn) on 2013-05-03
Changed in percona-toolkit:
assignee: nobody → Brian Fraser (fraserbn)
Brian Fraser (fraserbn) on 2013-05-03
Changed in percona-toolkit:
status: Confirmed → In Progress
Brian Fraser (fraserbn) wrote :

Steven: Many, many thanks for that .sql file. I had taken some stabs at this bug previously, but turns out I was looking for the wrong sort of problem.

The triage for this is simple: The "deep recursion" warning triggers if a sub recurses 100 times (that value is hardcoded in Perl, although greater in upcoming versions). The solution is also simple: Just don't recurse at all.

Changed in percona-toolkit:
milestone: none → 2.2.3
Changed in percona-toolkit:
assignee: Brian Fraser (fraserbn) → nobody
Changed in percona-toolkit:
status: In Progress → Triaged
Changed in percona-toolkit:
importance: Undecided → High
Daniel Nichter (daniel-nichter) wrote :

I can reproduce this now in 2.2.3:

Deep recursion on subroutine "SchemaIterator::_iterate_dbh" at bin/pt-table-checksum line 7398.

Steven Barre (steven-8) wrote :

Thanks! @daniel-nichter ! Tested on my systems and its working great now.

Daniel Nichter (daniel-nichter) wrote :

Thanks Steven, glad to hear it!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments