| 2012-04-11 16:43:54 |
Miguel Angel Nieto |
bug |
|
|
added bug |
| 2012-04-11 16:43:54 |
Miguel Angel Nieto |
attachment added |
|
patch to move the sleep section https://bugs.launchpad.net/bugs/979092/+attachment/3047646/+files/patch |
|
| 2012-04-11 16:45:29 |
Miguel Angel Nieto |
description |
for example, if you use --limit 1000 and --sleep 60 instead of fetch 1000 rows and then sleep 60 seconds, it sleeps 60 seconds every row. So for example purge processes needs lot of time:
pt-archiver \
--source h=localhost,D=test,t=t,u=root \
--purge \
--where 'i > 1' \
--limit 5 \
--sleep 5 \
--bulk-delete \
--pid=/tmp/mk-archiver-sessions.pid
# pt_archiver:4201 2597 Got another row in this chunk
# pt_archiver:4218 2597 Sleeping 5
# pt_archiver:4201 2597 Got another row in this chunk
# pt_archiver:4218 2597 Sleeping 5
# pt_archiver:4201 2597 Got another row in this chunk
# pt_archiver:4218 2597 Sleeping 5
# pt_archiver:4201 2597 Got another row in this chunk
# pt_archiver:4218 2597 Sleeping 5
# pt_archiver:4112 2597 No more rows in this chunk; doing bulk operations
# pt_archiver:4165 2597 Bulk deleted 5 rows
So, it deletes data in groups of 5 rows, but it stops after every fetched row. If I remove the sleep option:
pt-archiver \
--source h=localhost,D=test,t=t,u=root \
--purge \
--where 'i > 1' \
--limit 5 \
--bulk-delete \
--pid=/tmp/mk-archiver-sessions.pid
# pt_archiver:4201 2602 Got another row in this chunk
# pt_archiver:4201 2602 Got another row in this chunk
# pt_archiver:4201 2602 Got another row in this chunk
# pt_archiver:4201 2602 Got another row in this chunk
# pt_archiver:4112 2602 No more rows in this chunk; doing bulk operations
# pt_archiver:4165 2602 Bulk deleted 5 rows
To solve the problem I've moved the sleep section of the code to the end of the "This code is for the bulk archiving functionality" section.
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4112 2110 No more rows in this chunk; doing bulk operations
# pt_archiver:4165 2110 Bulk deleted 5 rows
# pt_archiver:4179 2110 Sleeping 5
# pt_archiver:4191 2110 Fetching rows in next chunk
# pt_archiver:4196 2110 Fetched 5 rows
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4112 2110 No more rows in this chunk; doing bulk operations
# pt_archiver:4165 2110 Bulk deleted 5 rows
# pt_archiver:4179 2110 Sleeping 5 |
for example, if you use --limit 1000 and --sleep 60 instead of fetching 1000 rows and then sleep 60 seconds, it sleeps 60 seconds every row. So for example purge processes needs lot of time:
pt-archiver \
--source h=localhost,D=test,t=t,u=root \
--purge \
--where 'i > 1' \
--limit 5 \
--sleep 5 \
--bulk-delete \
--pid=/tmp/mk-archiver-sessions.pid
# pt_archiver:4201 2597 Got another row in this chunk
# pt_archiver:4218 2597 Sleeping 5
# pt_archiver:4201 2597 Got another row in this chunk
# pt_archiver:4218 2597 Sleeping 5
# pt_archiver:4201 2597 Got another row in this chunk
# pt_archiver:4218 2597 Sleeping 5
# pt_archiver:4201 2597 Got another row in this chunk
# pt_archiver:4218 2597 Sleeping 5
# pt_archiver:4112 2597 No more rows in this chunk; doing bulk operations
# pt_archiver:4165 2597 Bulk deleted 5 rows
So, it deletes data in groups of 5 rows, but it stops after every fetched row. If I remove the sleep option:
pt-archiver \
--source h=localhost,D=test,t=t,u=root \
--purge \
--where 'i > 1' \
--limit 5 \
--bulk-delete \
--pid=/tmp/mk-archiver-sessions.pid
# pt_archiver:4201 2602 Got another row in this chunk
# pt_archiver:4201 2602 Got another row in this chunk
# pt_archiver:4201 2602 Got another row in this chunk
# pt_archiver:4201 2602 Got another row in this chunk
# pt_archiver:4112 2602 No more rows in this chunk; doing bulk operations
# pt_archiver:4165 2602 Bulk deleted 5 rows
To solve the problem I've moved the sleep section of the code to the end of the "This code is for the bulk archiving functionality" section.
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4112 2110 No more rows in this chunk; doing bulk operations
# pt_archiver:4165 2110 Bulk deleted 5 rows
# pt_archiver:4179 2110 Sleeping 5
# pt_archiver:4191 2110 Fetching rows in next chunk
# pt_archiver:4196 2110 Fetched 5 rows
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4211 2110 Got another row in this chunk
# pt_archiver:4112 2110 No more rows in this chunk; doing bulk operations
# pt_archiver:4165 2110 Bulk deleted 5 rows
# pt_archiver:4179 2110 Sleeping 5 |
|
| 2012-04-16 17:13:20 |
Daniel Nichter |
summary |
pt-archiver stops --sleep X seconds after fetching one row, not every --limit X rows |
pt-archiver --sleep conflicts with bulk operations |
|
| 2012-04-16 17:13:31 |
Daniel Nichter |
nominated for series |
|
percona-toolkit/2.0 |
|
| 2012-04-16 17:13:31 |
Daniel Nichter |
bug task added |
|
percona-toolkit/2.0 |
|
| 2012-04-16 17:13:31 |
Daniel Nichter |
nominated for series |
|
percona-toolkit/2.1 |
|
| 2012-04-16 17:13:31 |
Daniel Nichter |
bug task added |
|
percona-toolkit/2.1 |
|
| 2012-04-16 17:13:38 |
Daniel Nichter |
percona-toolkit/2.0: status |
New |
Confirmed |
|
| 2012-04-16 17:13:40 |
Daniel Nichter |
percona-toolkit/2.1: status |
New |
Confirmed |
|
| 2012-04-16 17:14:03 |
Daniel Nichter |
tags |
|
pt-archiver sleep |
|
| 2012-04-17 22:31:12 |
Daniel Nichter |
tags |
pt-archiver sleep |
percona-22758 pt-archiver sleep |
|
| 2012-04-17 22:33:23 |
Daniel Nichter |
percona-toolkit/2.1: status |
Confirmed |
In Progress |
|
| 2012-04-17 22:33:25 |
Daniel Nichter |
percona-toolkit/2.1: importance |
Undecided |
High |
|
| 2012-04-17 22:33:27 |
Daniel Nichter |
percona-toolkit/2.1: assignee |
|
Daniel Nichter (daniel-nichter) |
|
| 2012-04-17 22:33:32 |
Daniel Nichter |
percona-toolkit/2.1: milestone |
|
2.1.2 |
|
| 2012-04-17 22:55:57 |
Daniel Nichter |
branch linked |
|
lp:~percona-toolkit-dev/percona-toolkit/fix-sleep-bug-979092 |
|
| 2012-04-17 22:59:39 |
Daniel Nichter |
percona-toolkit/2.1: status |
In Progress |
Fix Committed |
|
| 2012-05-30 21:02:34 |
Daniel Nichter |
percona-toolkit/2.0: status |
Confirmed |
Fix Committed |
|
| 2012-05-30 21:02:35 |
Daniel Nichter |
percona-toolkit/2.0: importance |
Undecided |
Medium |
|
| 2012-05-30 21:02:37 |
Daniel Nichter |
percona-toolkit/2.0: assignee |
|
Daniel Nichter (daniel-nichter) |
|
| 2012-05-30 21:02:39 |
Daniel Nichter |
percona-toolkit/2.0: milestone |
|
2.0.5 |
|
| 2012-06-09 22:35:22 |
Daniel Nichter |
percona-toolkit/2.0: status |
Fix Committed |
Fix Released |
|
| 2012-06-19 19:58:06 |
Daniel Nichter |
percona-toolkit/2.1: status |
Fix Committed |
Fix Released |
|
