pt-table-checksum doesn't wait for checksum table to replicate

[Baron Schwartz]

I've observed nondeterministic behavior in this test:

[baron@localhost stabilize-test-suite]$ perl t/pt-table-checksum/skip_innodb.t
# Stopping/reconfiguring/restarting sandboxes 12348 and 12349
1..3
ok 1 - Ran without InnoDB (bug 996110)
ok 2 - 0 exit status (bug 996110)
# Shutting down sandboxes
ok 3 - Sandbox servers

[baron@localhost stabilize-test-suite]$ perl t/pt-table-checksum/skip_innodb.t
# Stopping/reconfiguring/restarting sandboxes 12348 and 12349
1..3
ok 1 - Ran without InnoDB (bug 996110)
not ok 2 - 0 exit status (bug 996110)
# Failed test '0 exit status (bug 996110)'
# in t/pt-table-checksum/skip_innodb.t at line 61.
# got: '1'
# expected: '0'
# Shutting down sandboxes
ok 3 - Sandbox servers
# Looks like you failed 1 test of 3.

[baron@localhost stabilize-test-suite]$ perl t/pt-table-checksum/skip_innodb.t
# Stopping/reconfiguring/restarting sandboxes 12348 and 12349
1..3
ok 1 - Ran without InnoDB (bug 996110)
ok 2 - 0 exit status (bug 996110)
# Shutting down sandboxes
ok 3 - Sandbox servers

When I saved the output of test 2 to a file in /tmp, I found a lot of the following:

06-04T09:01:26 2 0 0 1 0 0.008 mysql.time_zone_name
06-04T09:01:26 Error waiting for the last checksum of table mysql.time_zone_transition to replicate to replica localhost.localdomain: DBD::mysql::db selectrow_array failed: Table 'percona.checksums' doesn't exist [for Statement "SELECT MAX(chunk) FROM `percona`.`checksums` WHERE db='mysql' AND tbl='time_zone_transition' AND master_crc IS NOT NULL"] at /home/baron/stabilize-test-suite//bin/pt-table-checksum line 7513.

Check that the replica is running and has the replicate table `percona`.`checksums`. Checking the replica for checksum differences will probably cause another error.
06-04T09:01:26 Error checksumming table mysql.time_zone_transition: Use of uninitialized value in numeric lt (<) at /home/baron/stabilize-test-suite//bin/pt-table-checksum line 7533.

This hints to me that pt-table-checksum needs to do something smarter. After creating the checksum table, it needs to wait until this table appears on all of the replicas it's detected. In addition, we need to fix the error on line 7533.

[Daniel Nichter (daniel-nichter)]

This bug is not Fix Committed until lp:~percona-toolkit-dev/percona-toolkit/stabilize-test-suite is merged into lp:percona-toolkit/2.1

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PT-316