Print timestamp for all xtrabackup messages

[Vadim Tkachenko]

Looking on compact backup --rebuild-indexes process:

Rebuilding indexes for table tpcc/customer (space id: 20)
  Found index idx_customer
  Dropping 1 index(es).
  Rebuilding 1 index(es).
Rebuilding indexes for table tpcc/district (space id: 19)
Rebuilding indexes for table tpcc/history (space id: 22)
  Found index fkey_history_1
  Found index fkey_history_2
  Dropping 2 index(es).
  Rebuilding 2 index(es).
Rebuilding indexes for table tpcc/item (space id: 17)
Rebuilding indexes for table tpcc/new_orders (space id: 23)
Rebuilding indexes for table tpcc/order_line (space id: 26)
  Found index fkey_order_line_2
  Dropping 1 index(es).
  Rebuilding 1 index(es).

I wonder:
1. Can we print timestamp before each "Rebuilding indexes", just to know how long it take to rebuild.
2. Perform rebuilds in N parallel threads. I.e. table tpcc/order_line is rebuilt in Thread 1, and table tpcc/item in Thread 2, etc.

[Alexey Kopytov (akopytov)]

1. Sure.

2. xtrabackup calls InnoDB's row_merge_build_indexes() which accepts multiple index definitions, then does the following:

a) scans the clustered index, and writes secondary index tuples into temporary files, one file per index
b) sorts index tuples from temporary files and populates index pages. This handles one index at a time.

So it would be possible to break a single row_merge_build_indexes() call into many calls (one per index) and do it in parallel. But that would also mean that the clustered index would be scanned multiple times.

One way to fix this is to make fast index creation parallel in the server, so that step b) is performed with a specified number of threads. That would also provide the requested functionality after rebasing XB on the corresponding PS version.

[Alexey Kopytov (akopytov)]

Created https://blueprints.launchpad.net/percona-server/+spec/parallel-fast-index-creation and https://blueprints.launchpad.net/percona-xtrabackup/+spec/parallel-index-rebuild to track #2.

So this bug report is about #1 (i.e. print timestamp for all xtrabackup messages) which has a more general scope than compact backups, updating the summary accordingly.

[Vadim Tkachenko (vadim-tk)]

Alexey,

#2 - probably then we need to rebuild indexes in parallel on different tables
using row_merge_build_indexes()

On Thu, Jan 31, 2013 at 4:31 AM, Alexey Kopytov
<email address hidden> wrote:
> 1. Sure.
>
> 2. xtrabackup calls InnoDB's row_merge_build_indexes() which accepts
> multiple index definitions, then does the following:
>
> a) scans the clustered index, and writes secondary index tuples into temporary files, one file per index
> b) sorts index tuples from temporary files and populates index pages. This handles one index at a time.
>
> So it would be possible to break a single row_merge_build_indexes() call
> into many calls (one per index) and do it in parallel. But that would
> also mean that the clustered index would be scanned multiple times.
>
> One way to fix this is to make fast index creation parallel in the
> server, so that step b) is performed with a specified number of threads.
> That would also provide the requested functionality after rebasing XB on
> the corresponding PS version.
>
> --
> You received this bug notification because you are a member of Percona
> core, which is subscribed to Percona XtraBackup.
> https://bugs.launchpad.net/bugs/1109241
>
> Title:
> Compact backup improvements
>
> Status in Percona XtraBackup:
> New
>
> Bug description:
> Looking on compact backup --rebuild-indexes process:
>
> Rebuilding indexes for table tpcc/customer (space id: 20)
> Found index idx_customer
> Dropping 1 index(es).
> Rebuilding 1 index(es).
> Rebuilding indexes for table tpcc/district (space id: 19)
> Rebuilding indexes for table tpcc/history (space id: 22)
> Found index fkey_history_1
> Found index fkey_history_2
> Dropping 2 index(es).
> Rebuilding 2 index(es).
> Rebuilding indexes for table tpcc/item (space id: 17)
> Rebuilding indexes for table tpcc/new_orders (space id: 23)
> Rebuilding indexes for table tpcc/order_line (space id: 26)
> Found index fkey_order_line_2
> Dropping 1 index(es).
> Rebuilding 1 index(es).
>
> I wonder:
> 1. Can we print timestamp before each "Rebuilding indexes", just to know how long it take to rebuild.
> 2. Perform rebuilds in N parallel threads. I.e. table tpcc/order_line is rebuilt in Thread 1, and table tpcc/item in Thread 2, etc.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/percona-xtrabackup/+bug/1109241/+subscriptions

--
Vadim Tkachenko, CTO, Percona
Phone [phone], Skype: vadimtk153
Schedule meeting: http://meetme.so/VadimTkachenko

Join us in Santa Clara for the annual Percona Live MySQL Conference & Expo 2013!
http://www.percona.com/live/mysql-conference-2013/

Looking for Replication with Data Consistency?
Try Percona XtraDB Cluster!

[Alexey Kopytov (akopytov)]

Hi Vadim,

On Fri, 01 Feb 2013 16:38:43 -0000, Vadim Tkachenko wrote:
> Alexey,
>
> #2 - probably then we need to rebuild indexes in parallel on different tables
> using row_merge_build_indexes()
>
Yes, that is possible. Created
https://blueprints.launchpad.net/percona-xtrabackup/+spec/parallel-table-rebuild

[Roel Van de Paar (roel11)]

Adding vote for this bug to be implemented - from a QA perspective it is very handy to see duration of events to see if we get any runaways. Thanks.

[Roel Van de Paar (roel11)]

s/bug/feature request/

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-985