xbstream sometimes fails while extracting

[Steven Ayre]

I collected a large streaming backup over SSH which was written to a file (named 20120406_092210_full.xbs)

When extracting the file, I found xbstream intermittently fails to read from the file. The error is:
xb_stream_read_chunk(): unexpected end of stream at offset 0xf4c3fffc.

The backup command was:
innobackupex --user=$USER --password=$PASS --compress --compress-threads=2 --stream=xbstream /home/$USER/mysqlbackupdir

STDOUT was piped over SSH and written to the .xbs file.

The extraction command was:
xbstream -x -v < ../20120406_092210_full.xbs

This appears to be a bug in xbstream, not corrupt data. The offset was different every time the message appeared, and eventually the file extracted without any error, creating all the expected files.

Platform is Debian Squeeze x64 using the xtrabackup dotdebs from percona.com

[Steven Ayre (steveayre)]

I've just had smartd report Ofline uncorrectable sectors on one of the drives on this server, so this could be a HDD issue not xbstream issue.

[Stewart Smith (stewart)]

On Tue, 10 Apr 2012 12:32:32 -0000, Steven Ayre <email address hidden> wrote:
> The extraction command was:
> xbstream -x -v < ../20120406_092210_full.xbs
>
> This appears to be a bug in xbstream, not corrupt data. The offset was
> different every time the message appeared, and eventually the file
> extracted without any error, creating all the expected files.

Thanks very much for the good bug report. We'll try to repeat and then fix.

--
Stewart Smith

[Alexey Kopytov (akopytov)]

The workaround is to first save the resulting stream to a file, and then extract it in a separate step:

innobackupex --stream=xbstream --compress /tmp | ssh user@host "cat - > /tmp/backup.xbstream"
(on remote host):
xbstream -xv -C /data/backup < /tmp/backup.xbstream

[Steven Ayre (steveayre)]

Hi Alexey. Any update on this?

The workaround does not work for me - as stated originally I have already streamed it over SSH to a file. The extraction command is reading directly from the saved file:
xbstream -x -v < ../20120406_092210_full.xbs

Having run strace it appears that it attempts to read the XBSTCK01 data as 14 bytes. Usually read() returns exactly 14 bytes:
read(0, "XBSTCK01\0P\n\0\0\0", 14) = 14

When the error occurs the last read was shorter than the expected 14 bytes (even though there is more data available). It appears that xbstream doesn't handle read() returning less data than requested.
read(0, "XBSTCK01\0P\n", 14) = 11

[Steven Ayre (steveayre)]

Nevermind - just seen the fix released tag. I will test with trunk. :)

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-326