Comment 173 for bug 1470250

Revision history for this message
Emsi (trash1-z) wrote :

Bad news.
I tried to replicate the test results on my own.
I downloaded and installed the first kernel marked as good:
# uname -a
Linux backup-01 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
I performed the tiobench and simultaneity the backup.
At first it looked like the is no corruption as nothing worrying was reported in the dmesg. Unfortunately the OS behavior indicates that the there is at least memory corruption:

root@backup-01:~# dpkg-reconfigure --frontend noninteractive tzdata
Can't load '/usr/lib/perl/5.18/auto/IO/IO.so' for module IO: /usr/lib/perl/5.18/auto/IO/IO.so: failed to map segment from shared object: Cannot allocate memory at /usr/share/perl/5.18/XSLoader.pm line 68.
 at /usr/lib/perl/5.18/IO.pm line 11.
Compilation failed in require at /usr/lib/perl/5.18/IO/Handle.pm line 9.
BEGIN failed--compilation aborted at /usr/lib/perl/5.18/IO/Handle.pm line 9.
Compilation failed in require at /usr/lib/perl/5.18/IO/Seekable.pm line 9.
BEGIN failed--compilation aborted at /usr/lib/perl/5.18/IO/Seekable.pm line 9.
Compilation failed in require at /usr/lib/perl/5.18/IO/File.pm line 11.
BEGIN failed--compilation aborted at /usr/lib/perl/5.18/IO/File.pm line 11.
Compilation failed in require at /usr/share/perl/5.18/FileHandle.pm line 9.
Compilation failed in require at /usr/share/perl5/Debconf/Template.pm line 8.
BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Template.pm line 8.
Compilation failed in require at /usr/share/perl5/Debconf/Question.pm line 8.
BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Question.pm line 8.
Compilation failed in require at /usr/share/perl5/Debconf/Config.pm line 7.
BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Config.pm line 7.
Compilation failed in require at /usr/share/perl5/Debconf/Log.pm line 10.
Compilation failed in require at /usr/share/perl5/Debconf/Db.pm line 7.
BEGIN failed--compilation aborted at /usr/share/perl5/Debconf/Db.pm line 7.
Compilation failed in require at /usr/sbin/dpkg-reconfigure line 11.
BEGIN failed--compilation aborted at /usr/sbin/dpkg-reconfigure line 11.

I observed some random segfaults as well:
[ 7773.234696] tiotest[15052]: segfault at 0 ip 000000000040183e sp 00007f2ed15a3f00 error 4 in tiotest[400000+8000]
[ 7774.185489] apt-get[15073]: segfault at ffffffffffffffff ip 00007f2568a3bbfb sp 00007fff52a88840 error 5 in libapt-pkg.so.4.12.0[7f25689e5000+145000]

After reboot it suddenly started to work:

# dpkg-reconfigure --frontend noninteractive tzdata

Current default time zone: 'SystemV/CST6CDT'
Local time is now: Thu Apr 14 07:06:20 CDT 2016.
Universal Time is now: Thu Apr 14 12:06:20 UTC 2016.

That suggests that we're dealing with memory corruption bug that eventually leads to fs corruption.