Stream decryption fails with options in my.cnf

Bug #1190335 reported by Raghavendra D Prabhu
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
Low
Sergei Glushchenko
2.1
Won't Fix
Low
Unassigned
2.2
Won't Fix
Low
Unassigned
2.3
Fix Released
Low
Sergei Glushchenko

Bug Description

Following fails:
=================================================================

sudo innobackupex --defaults-file=/etc/my.cnf --socket=/var/lib/mysql/mysql.sock --galera-info --stream=xbstream --user=root --password=test /tmp | xbcrypt -d --encrypt-algo=AES256 --encrypt-key=6F3AD9F428143F133FD7D50D77D91EA4 > /dev/shm/test4

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Ireland Ltd 2009-2012. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

130613 00:24:47 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'root' (using password: YES).
130613 00:24:47 innobackupex: Connected to MySQL server
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql server version 5.5.30-log

innobackupex: Created backup directory /tmp
xbcrypt:xb_crypt_read_chunk: wrong chunk magic at offset 0x0.

130613 00:24:47 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp --tmpdir=/tmp --stream=xbstream
innobackupex: Waiting for ibbackup (pid=3778) to suspend
innobackupex: Suspend file '/tmp/xtrabackup_suspended_2'

xtrabackup_55 version 2.1.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: 608)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 536870912
xtrabackup: using O_DIRECT
130613 0:24:47 InnoDB: Warning: allocated tablespace 10, old maximum was 9
>> log scanned up to (9615410)
[01] Compressing, encrypting and streaming ./ibdata1
xtrabackup_55: Error writing file 'UNOPENED' (Errcode: 32)
encrypt: write to the destination file failed.
xb_stream_write_data() failed.
compress: write to the destination stream failed.
>> log scanned up to (9615410)
>> log scanned up to (9615410)
^C%

===========================================================

my.cnf - http://sprunge.us/YBPE

Tags: pxc
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Note,

sudo innobackupex --defaults-file=/etc/my.cnf --encrypt=AES256 --encrypt-key=6F3AD9F428143F133FD7D50D77D91EA4 --socket=/var/lib/mysql/mysql.sock --galera-info --stream=xbstream --user=
root --password=test /tmp | xbcrypt -d --encrypt-algo=AES256 --encrypt-key=6F3AD9F428143F133FD7D50D77D91EA4

works though.

So, it may be something wrong with parsing of my.cnf

Interestingly.

xtrabackup_55 --print-defaults
xtrabackup_55 would have been started with the following arguments:
--datadir=/var/lib/mysql --log_slow_queries=/var/lib/mysql/mysql-slow.log --long_query_time=10 --loose-log_slow_verbosity=microtime,query_plan,innodb,profiling,profiling_use_getrusage --loose-slow_query_log_timest
amp_always=true --loose-slow_query_log_timestamp_precision=microsecond --loose-slow_query_log_use_global_control=log_slow_filter,log_slow_rate_limit,log_slow_verbosity,long_query_time,min_examined_row_limit --loose-log_slow_slave_statements --log_slave_updates --server-id=908 --log_bin=/var/lib/mysql/mysql-bin.log --max_binlog_size=100M --expire_logs_days=2 --binlog_format=ROW --thread_stack=256K --thread_cache_size=512 --tmp_table_size=32M --max_heap_table_size=32M --max_connections=10000 --open-files-limit=65535 --table_open_cache=8192 --table_definition_cache=8192 --key_buffer_size=64M --innodb_buffer_pool_size=500M --innodb_flush_log_at_trx_commit=2 --innodb_flush_method=O_DIRECT --innodb_log_files_in_group=2 --innodb_log_file_size=512M --innodb_file_per_table=1 --loose-query_response_time_stats --wsrep_cluster_address=gcomm:// --wsrep_provider=/usr/lib64/libgalera_smm.so --wsrep_slave_threads=2 --wsrep_cluster_name=PXC --wsrep_sst_method=xtrabackup --wsrep_sst_auth=root:test --wsrep_node_name=Pxc1 --innodb_locks_unsafe_for_binlog=1 --innodb_autoinc_lock_mode=2 --compress --compact --encrypt=AES256 --encrypt-key=6F3AD9F428143F133FD7D50D77D91EA4

shows correctly.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

lp:1190343 depends on this. (PXC uses encryption for SST)

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Actually, I see another error even with parameters provided through innobackupex:

xbcrypt:xb_crypt_read_chunk: failed to read 65536 bytes for chunk payload at offset 0x1ec.

is the error on receiving side.

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Ireland Ltd 2009-2012. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

130613 01:37:12 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'root' (using pass
word: YES).
130613 01:37:12 innobackupex: Connected to MySQL server
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql server version 5.5.30-log

innobackupex: Created backup directory /tmp

130613 01:37:12 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp --tmpdir=/tmp --encrypt=AES256 --e
ncrypt-key=6F3AD9F428143F133FD7D50D77D91EA4 --encrypt-threads=1 --stream=xbstream
innobackupex: Waiting for ibbackup (pid=6175) to suspend
innobackupex: Suspend file '/tmp/xtrabackup_suspended_2'

xtrabackup_55 version 2.1.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: 608)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 536870912
xtrabackup: using O_DIRECT
130613 1:37:12 InnoDB: Warning: allocated tablespace 10, old maximum was 9
>> log scanned up to (9616286)
[01] Compressing, encrypting and streaming ./ibdata1
^Gxtrabackup_55: Error writing file 'UNOPENED' (Errcode: 32)
encrypt: write to the destination file failed.
xb_stream_write_data() failed.
compress: write to the destination stream failed.
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)

is the error on sending side.

I tried with https://bazaar.launchpad.net/~gl-az/percona-xtrabackup/BT23557-2.1-lp1160778/download/head:/innobackup1.5.1-20090305061108-fxjvmhr914de7q81-1/innobackupex

from commit of https://bugs.launchpad.net/percona-xtrabackup/+bug/1160778

but it has not helped.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

One more thing, if it helps,

backup-my.cnf was transferred correctly, only after that it failed (as the offset explains).

Revision history for this message
Alexey Kopytov (akopytov) wrote :

George, can you analyze this?

Changed in percona-xtrabackup:
assignee: nobody → George Ormond Lorch III (gl-az)
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

Since #3 involves a different stage (but part of same process), I will report a separate bug for this.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :
Revision history for this message
George Ormond Lorch III (gl-az) wrote : Re: [Bug 1190335] Re: Stream decryption fails with options in my.cnf

Yes, we had an IRC conversation yesterday that lead to this being
entered. Since X w/ encryption is now to be used with PXB SST we will
need to fix this as well as make sure to create a suite test case for
the xtrabackup use portion of it since no case exists that does
"innobackupex --stream=xbstream --encrypt... | xbcrypt -d ... | xbstream
-x ..." and no case that uses encryption options for innobackupex via
the .cnf file. So all of this needs a new case and whatever is causing
this issue obviously needs fixed too :)

On 6/13/2013 6:38 AM, Alexey Kopytov wrote:
> George, can you analyze this?
>
> ** Changed in: percona-xtrabackup
> Assignee: (unassigned) => George Ormond Lorch III (gl-az)
>

--
George O. Lorch III
Software Engineer, Percona
+1-888-401-3401 x542 US/Arizona (GMT -7)
skype: george.ormond.lorch.iii

Changed in percona-xtrabackup:
status: New → Incomplete
status: Incomplete → New
Revision history for this message
George Ormond Lorch III (gl-az) wrote :

OK, this is failing because only xtrabackup is encrypting because it is finding the encryption options within the config file while innobackupex is not looking for any encryption options in the config file. The resulting stream is useless because it is only partially encrypted and thus xbcrypt -d is seeing the unencrypted data as unrecognizable block headers.

Revision history for this message
Alexey Kopytov (akopytov) wrote :

This will be fixed when https://blueprints.launchpad.net/percona-xtrabackup/+spec/rewrite-innobackupex-in-c is implemented. Requested documenting this limitation as bug #1193208.

tags: added: pxc
Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

An implication of this is that encryption options should be
provided to both innobackupex and xtrabackup, ie. there is no way
to avoid encrypted backup from innobackupex if it is already
specified under [xtrabackup] in my.cnf. This can be problematic
from a SST point of view., will need to add possible workarounds
for this.

Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-135

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.