xbstream sometimes fails while extracting

Reported by Steven Ayre on 2012-04-10
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup
High
Alexey Kopytov
2.0
High
Alexey Kopytov
2.1
High
Alexey Kopytov

Bug Description

I collected a large streaming backup over SSH which was written to a file (named 20120406_092210_full.xbs)

When extracting the file, I found xbstream intermittently fails to read from the file. The error is:
xb_stream_read_chunk(): unexpected end of stream at offset 0xf4c3fffc.

The backup command was:
innobackupex --user=$USER --password=$PASS --compress --compress-threads=2 --stream=xbstream /home/$USER/mysqlbackupdir

STDOUT was piped over SSH and written to the .xbs file.

The extraction command was:
xbstream -x -v < ../20120406_092210_full.xbs

This appears to be a bug in xbstream, not corrupt data. The offset was different every time the message appeared, and eventually the file extracted without any error, creating all the expected files.

Platform is Debian Squeeze x64 using the xtrabackup dotdebs from percona.com

Related branches

Steven Ayre (steveayre) wrote :

I've just had smartd report Ofline uncorrectable sectors on one of the drives on this server, so this could be a HDD issue not xbstream issue.

On Tue, 10 Apr 2012 12:32:32 -0000, Steven Ayre <email address hidden> wrote:
> The extraction command was:
> xbstream -x -v < ../20120406_092210_full.xbs
>
> This appears to be a bug in xbstream, not corrupt data. The offset was
> different every time the message appeared, and eventually the file
> extracted without any error, creating all the expected files.

Thanks very much for the good bug report. We'll try to repeat and then fix.

--
Stewart Smith

Alexey Kopytov (akopytov) wrote :

The workaround is to first save the resulting stream to a file, and then extract it in a separate step:

innobackupex --stream=xbstream --compress /tmp | ssh user@host "cat - > /tmp/backup.xbstream"
(on remote host):
xbstream -xv -C /data/backup < /tmp/backup.xbstream

Changed in percona-xtrabackup:
status: New → In Progress
assignee: nobody → Alexey Kopytov (akopytov)
Steven Ayre (steveayre) wrote :

Hi Alexey. Any update on this?

The workaround does not work for me - as stated originally I have already streamed it over SSH to a file. The extraction command is reading directly from the saved file:
xbstream -x -v < ../20120406_092210_full.xbs

Having run strace it appears that it attempts to read the XBSTCK01 data as 14 bytes. Usually read() returns exactly 14 bytes:
read(0, "XBSTCK01\0P\n\0\0\0", 14) = 14

When the error occurs the last read was shorter than the expected 14 bytes (even though there is more data available). It appears that xbstream doesn't handle read() returning less data than requested.
read(0, "XBSTCK01\0P\n", 14) = 11

Steven Ayre (steveayre) wrote :

Nevermind - just seen the fix released tag. I will test with trunk. :)

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers