Stream decryption fails with options in my.cnf

[Raghavendra D Prabhu]

Following fails:
=================================================================

sudo innobackupex --defaults-file=/etc/my.cnf --socket=/var/lib/mysql/mysql.sock --galera-info --stream=xbstream --user=root --password=test /tmp | xbcrypt -d --encrypt-algo=AES256 --encrypt-key=6F3AD9F428143F133FD7D50D77D91EA4 > /dev/shm/test4

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Ireland Ltd 2009-2012. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

130613 00:24:47 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'root' (using password: YES).
130613 00:24:47 innobackupex: Connected to MySQL server
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql server version 5.5.30-log

innobackupex: Created backup directory /tmp
xbcrypt:xb_crypt_read_chunk: wrong chunk magic at offset 0x0.

130613 00:24:47 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp --tmpdir=/tmp --stream=xbstream
innobackupex: Waiting for ibbackup (pid=3778) to suspend
innobackupex: Suspend file '/tmp/xtrabackup_suspended_2'

xtrabackup_55 version 2.1.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: 608)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 536870912
xtrabackup: using O_DIRECT
130613 0:24:47 InnoDB: Warning: allocated tablespace 10, old maximum was 9
>> log scanned up to (9615410)
[01] Compressing, encrypting and streaming ./ibdata1
xtrabackup_55: Error writing file 'UNOPENED' (Errcode: 32)
encrypt: write to the destination file failed.
xb_stream_write_data() failed.
compress: write to the destination stream failed.
>> log scanned up to (9615410)
>> log scanned up to (9615410)
^C%

===========================================================

my.cnf - http://sprunge.us/YBPE

[Raghavendra D Prabhu (raghavendra-prabhu)]

Note,

sudo innobackupex --defaults-file=/etc/my.cnf --encrypt=AES256 --encrypt-key=6F3AD9F428143F133FD7D50D77D91EA4 --socket=/var/lib/mysql/mysql.sock --galera-info --stream=xbstream --user=
root --password=test /tmp | xbcrypt -d --encrypt-algo=AES256 --encrypt-key=6F3AD9F428143F133FD7D50D77D91EA4

works though.

So, it may be something wrong with parsing of my.cnf

Interestingly.

xtrabackup_55 --print-defaults
xtrabackup_55 would have been started with the following arguments:
--datadir=/var/lib/mysql --log_slow_queries=/var/lib/mysql/mysql-slow.log --long_query_time=10 --loose-log_slow_verbosity=microtime,query_plan,innodb,profiling,profiling_use_getrusage --loose-slow_query_log_timest
amp_always=true --loose-slow_query_log_timestamp_precision=microsecond --loose-slow_query_log_use_global_control=log_slow_filter,log_slow_rate_limit,log_slow_verbosity,long_query_time,min_examined_row_limit --loose-log_slow_slave_statements --log_slave_updates --server-id=908 --log_bin=/var/lib/mysql/mysql-bin.log --max_binlog_size=100M --expire_logs_days=2 --binlog_format=ROW --thread_stack=256K --thread_cache_size=512 --tmp_table_size=32M --max_heap_table_size=32M --max_connections=10000 --open-files-limit=65535 --table_open_cache=8192 --table_definition_cache=8192 --key_buffer_size=64M --innodb_buffer_pool_size=500M --innodb_flush_log_at_trx_commit=2 --innodb_flush_method=O_DIRECT --innodb_log_files_in_group=2 --innodb_log_file_size=512M --innodb_file_per_table=1 --loose-query_response_time_stats --wsrep_cluster_address=gcomm:// --wsrep_provider=/usr/lib64/libgalera_smm.so --wsrep_slave_threads=2 --wsrep_cluster_name=PXC --wsrep_sst_method=xtrabackup --wsrep_sst_auth=root:test --wsrep_node_name=Pxc1 --innodb_locks_unsafe_for_binlog=1 --innodb_autoinc_lock_mode=2 --compress --compact --encrypt=AES256 --encrypt-key=6F3AD9F428143F133FD7D50D77D91EA4

shows correctly.

[Raghavendra D Prabhu (raghavendra-prabhu)]

lp:1190343 depends on this. (PXC uses encryption for SST)

[Raghavendra D Prabhu (raghavendra-prabhu)]

Actually, I see another error even with parameters provided through innobackupex:

xbcrypt:xb_crypt_read_chunk: failed to read 65536 bytes for chunk payload at offset 0x1ec.

is the error on receiving side.

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona Ireland Ltd 2009-2012. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

130613 01:37:12 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_file=/etc/my.cnf;mysql_read_default_group=xtrabackup;mysql_socket=/var/lib/mysql/mysql.sock' as 'root' (using pass
word: YES).
130613 01:37:12 innobackupex: Connected to MySQL server
IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex: Using mysql server version 5.5.30-log

innobackupex: Created backup directory /tmp

130613 01:37:12 innobackupex: Starting ibbackup with command: xtrabackup_55 --defaults-file="/etc/my.cnf" --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/tmp --tmpdir=/tmp --encrypt=AES256 --e
ncrypt-key=6F3AD9F428143F133FD7D50D77D91EA4 --encrypt-threads=1 --stream=xbstream
innobackupex: Waiting for ibbackup (pid=6175) to suspend
innobackupex: Suspend file '/tmp/xtrabackup_suspended_2'

xtrabackup_55 version 2.1.3 for Percona Server 5.5.16 Linux (x86_64) (revision id: 608)
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: Target instance is assumed as followings.
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 536870912
xtrabackup: using O_DIRECT
130613 1:37:12 InnoDB: Warning: allocated tablespace 10, old maximum was 9
>> log scanned up to (9616286)
[01] Compressing, encrypting and streaming ./ibdata1
^Gxtrabackup_55: Error writing file 'UNOPENED' (Errcode: 32)
encrypt: write to the destination file failed.
xb_stream_write_data() failed.
compress: write to the destination stream failed.
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)
>> log scanned up to (9616286)

is the error on sending side.

I tried with https://bazaar.launchpad.net/~gl-az/percona-xtrabackup/BT23557-2.1-lp1160778/download/head:/innobackup1.5.1-20090305061108-fxjvmhr914de7q81-1/innobackupex

from commit of https://bugs.launchpad.net/percona-xtrabackup/+bug/1160778

but it has not helped.

[Raghavendra D Prabhu (raghavendra-prabhu)]

One more thing, if it helps,

backup-my.cnf was transferred correctly, only after that it failed (as the offset explains).

[Alexey Kopytov (akopytov)]

George, can you analyze this?

[Raghavendra D Prabhu (raghavendra-prabhu)]

Since #3 involves a different stage (but part of same process), I will report a separate bug for this.

[Raghavendra D Prabhu (raghavendra-prabhu)]

Reported that here https://bugs.launchpad.net/percona-xtrabackup/+bug/1190610

[George Ormond Lorch III (gl-az)]

Yes, we had an IRC conversation yesterday that lead to this being
entered. Since X w/ encryption is now to be used with PXB SST we will
need to fix this as well as make sure to create a suite test case for
the xtrabackup use portion of it since no case exists that does
"innobackupex --stream=xbstream --encrypt... | xbcrypt -d ... | xbstream
-x ..." and no case that uses encryption options for innobackupex via
the .cnf file. So all of this needs a new case and whatever is causing
this issue obviously needs fixed too :)

On 6/13/2013 6:38 AM, Alexey Kopytov wrote:
> George, can you analyze this?
>
> ** Changed in: percona-xtrabackup
> Assignee: (unassigned) => George Ormond Lorch III (gl-az)
>

--
George O. Lorch III
Software Engineer, Percona
[phone] x542 US/Arizona (GMT -7)
skype: george.ormond.lorch.iii

[George Ormond Lorch III (gl-az)]

OK, this is failing because only xtrabackup is encrypting because it is finding the encryption options within the config file while innobackupex is not looking for any encryption options in the config file. The resulting stream is useless because it is only partially encrypted and thus xbcrypt -d is seeing the unencrypted data as unrecognizable block headers.

[Alexey Kopytov (akopytov)]

This will be fixed when https://blueprints.launchpad.net/percona-xtrabackup/+spec/rewrite-innobackupex-in-c is implemented. Requested documenting this limitation as bug #1193208.

[Raghavendra D Prabhu (raghavendra-prabhu)]

An implication of this is that encryption options should be
provided to both innobackupex and xtrabackup, ie. there is no way
to avoid encrypted backup from innobackupex if it is already
specified under [xtrabackup] in my.cnf. This can be problematic
from a SST point of view., will need to add possible workarounds
for this.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-135