e2fsck crashed with SIGSEGV in strlen()

[ingo]

Binary package hint: e2fsprogs

I just tried to 'repair' (in fact only check) a ext3 filesystem as created by QNAP TS-209 NAS for the data-partitition:

e2fsck -n /dev/sdc3
e2fsck 1.40.8 (13-Mar-2008)
/dev/sdc3 has unsupported feature(s):Segmentation fault (core dumped)

That's all.

ProblemType: Crash
Architecture: amd64
Date: Mon Aug 11 20:12:56 2008
DistroRelease: Ubuntu 8.04
ExecutablePath: /sbin/e2fsck
NonfreeKernelModules: nvidia
Package: e2fsprogs 1.40.8-2ubuntu2
PackageArchitecture: amd64
ProcCmdline: e2fsck -n /dev/sdc3
ProcEnviron:
 SHELL=/bin/bash
 PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
 LANG=de_DE.UTF-8
Signal: 11
SourcePackage: e2fsprogs
StacktraceTop:
 strlen () from /lib/libc.so.6
 vfprintf () from /lib/libc.so.6
 ?? () from /lib/libc.so.6
 vfprintf () from /lib/libc.so.6
 fprintf () from /lib/libc.so.6
Title: e2fsck crashed with SIGSEGV in strlen()
Uname: Linux 2.6.24-19-generic x86_64
UserGroups: adm audio cdrom dip fax floppy fuse lpadmin plugdev root scanner tape

[ingo (ingo-steiner)]

Found a quick solution - will call it patch:

just use e2fsprogs 1.41.0 from Debian-Lenny and all is fine - see Output of handling and correcting

Result see in attached file QNAP-fsck.txt

[Apport retracing service (apport)]

StacktraceTop:strlen () from /lib/libc.so.6
vfprintf () from /lib/libc.so.6
buffered_vfprintf () from /lib/libc.so.6
vfprintf () from /lib/libc.so.6
fprintf () from /lib/libc.so.6

[Theodore Ts'o (tytso)]

Can you send the output of "dumpe2fs -h /dev/sdc3"?

Thanks!!

[ingo (ingo-steiner)]

root@pp:/home/ingo# dumpe2fs -h /dev/sdc3
dumpe2fs 1.40.8 (13-Mar-2008)
Filesystem volume name: <none>
Last mounted on: <not available>
Filesystem UUID: 2854e675-ba59-4084-b530-759a967b4ac2
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal filetype extents sparse_super
Default mount options: (none)
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 91439104
Block count: 182855824
Reserved block count: 0
Free blocks: 179953562
Free inodes: 91439082
First block: 0
Block size: 4096
Fragment size: 4096
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 16384
Inode blocks per group: 512
Filesystem created: Wed Jul 16 18:38:08 2008
Last mount time: Thu Jul 17 23:01:27 2008
Last write time: Mon Aug 11 21:06:43 2008
Mount count: 0
Maximum mount count: -1
Last checked: Mon Aug 11 21:06:43 2008
Check interval: 0 (<none>)
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 128
Journal inode: 8
Default directory hash: tea
Directory Hash Seed: 7d13e982-8d92-4778-86c1-3860109ae5f6
Journal backup: inode blocks
Journal size: 128M

===========================================
Remark:
it is no longer the same content of /dev/sda3, I tried to repair it under Lenny, however it still cannot be mounted and Hardy crashes the very same way as before on 'e2fsck'.

[ingo (ingo-steiner)]

Probably some other usefull information: trying with an older version of e2fstools (under openSUSE 10.2) resulted in following error-message (no crash!):

fsck.ext3 -n /dev/sdc3
e2fsck 1.39 (29-May-2006)
fsck.ext3: Das Gerät oder die Ressource ist belegt beim Versuch,
/dev/sdc3 zu öffnen
Filesystem mounted or opened exclusively by another program?

I translate the first line of output into English:
"device or resource is busy while trying to open /dev/sdc3"

Seems there is some flag or attribute set to prevent acces?

[Theodore Ts'o (tytso)]

This sounds like a different problem, right? You've already blown away the original contents of the disk, so it's going to be impossible for us to try to reproduce what was going on on the Ubuntu e2fpsrogs version 1.40.8.

As for the "device or resource is busy", that means something else has the device open for exclusive access; maybe another e2fsck process, or the filesystem is mounted, etc. That's a totally separate issue, and it's a safegaurd to protect against users who accidentally try to run fsck on a filesystem which is mounted by /etc/mtab is corrupted, or who had forgotten that they or someone else was running e2fsck in another window, etc.

[ingo (ingo-steiner)]

If said in my first post, that the file system was created by QNAP TS-209 NAS.
It is an ext3 file system, but with some patches from QNAP applied (as far as I know to speed up RAID1).

This filesystem cannot be mounted on standard Linux PC's - but it is definitely ext3!

And that's why I have run e2fsck on it - at least tried:

on openSUSE with e2fsprogs 1.39 the check (with -n option!) does not run, see post before, but it does not crash!

on Hardy with v1.40 e2fsck just crashes - that's why I reported the bug

on Lenny with 1.41 e2fsck runs through fine and displays the faulty things, and without the -n option also corrects the errors.

So conclusion in the order I have performed:
filesystem is corrupted
cannot be mounted
Hardy: e2fsck crashes
OpenSUSE e2fsck reports message and exits
Lenny reports the faulty things, is able to correct some, but not all.
Filesystem can still not be mounted - but that is another story and no reason to crash e2fsck

If you need prove I'll do copy you mtab or whatever you like to show that it is NOT mounted!
Ingo

[ingo (ingo-steiner)]

So, forgot to mention:

if you want to reproduce the crash, you need a corrupted filesystem like I have it here.

That neiter e2fsck can repair it - its my fault (don't need the data anyhow)
But that e2fsck in Hardy crashes instead of displaying a message and exits, is a bug!

The filesystem still is corrupt and despite Lenny did fix a lot, it still is nos usable.

And, only Hardy's e2fsck still crashes when I call it!

[Theodore Ts'o (tytso)]

On Wed, Aug 13, 2008 at 07:38:25PM -0000, ingo wrote:
> If said in my first post, that the file system was created by QNAP
> TS-209 NAS. It is an ext3 file system, but with some patches from
> QNAP applied (as far as I know to speed up RAID1).

Actually, it looks like it's an ext4 filesystem --- or at least, it's
ext3 with some pre-release version of ext4. The dumpe2fs program
shows that the extents feature is enabled, which is an ext4 feature.

dumpe2fs 1.40.8 (13-Mar-2008)
...
Filesystem features: has_journal filetype extents sparse_super
                                          ^^^^^^^

Given the lack of other features which are enabled, I'm guessing it's
an early pre-release version of ext4 --- and I'm wondering how early,
since we've fixed a lot of ext4 bugs lately. but thta's a problem for
the QNAP NAS folks, and hopefully they will be tracking the latest
kernels and taking the latest ext4 updates and bug fixes.

Be that as it may, it's surprising that e2sck is crashing here, given
that (a) dumpe2fs 1.40.8 was able to display the filesystem feature
set, and (b) the code in e2fsck does substantially the same thing, and
that hasn't changed much between 1.40.8 and 1.41.0.

In any case, given the extents feature, you are not going to be able
to mount it using a stock Ubuntu kernel, since last I checked it
doesn't enable ext4 by default. It may be that an Ubuntu Interpid
kernel has the correct support, but you'll probably need to do the
following before it will mount.

1) Install e2fsprogs 1.41.0

2) Run "tune2fs -E test_fs /dev/sdc3"

3) Mount it via "mount -t ext4dev /dev/sdc3 /mnt"

OK, so let's review the bidding here.

> filesystem is corrupted
> cannot be mounted

Not necessarily; this is an ext4 filesystem, so unless you have the
ext4 filesystem compiled into your kernel, (and given Ubuntu's
unfortunate choice of vol_id instead of blkid libraries, you have to
explicitly tell mount to use the ext4 module via "mount -t ext4dev" ),
it's not mountable as an ext3 filesystem.

> Hardy: e2fsck crashes

That's wierd. I think we'll need to run it under a debugger and see
if we can get a useful stack trace out of it. Possibly we'll need to
do this using an unstripped binary or a recompiled-from-scratch
binary.

With the latest Lenny packages e2fsprogs now has debug packages, but
unfortunately not the version that Ubuntu Hardy is shipping with.

> OpenSUSE e2fsck reports message and exits

That's also wierd. I'm not sure what's going on there.

> Lenny reports the faulty things, is able to correct some, but not all.

You haven't uploaded what Lenny's e2fsck has reported, so I can't
really evaluate it. That also might be because whatever QNAS has
cooked up is non-standard and an older pre-release version of ext4,
and Lenny is shipping with e2fsprogs 1.41.0, which is designed to work
with the latest version of ext4 as can be found in the
mainline kernels.

I hope this is helpful,

     - Ted

[ingo (ingo-steiner)]

Many thanks Ted,

I think you hit the point!
For sure lenny did not 'repair' the file system, but left it in an unusable condition. I was able to even recover some of the filesystem (root directories9 by using 'debugfs' 'open /dev/sdc3' ...

So I now know where to go and I discard the unusable partitition. But I will get my NAS back and repaired soon (I hope) and then be able to create a 'virgin QNAP ext3' and check with Lenny or even Intrepid whether it is a ext4-alpha!

So for the moment, dont't worry about the ext4 story, just fsck.ext3 should not crash when operating on such a 'homebrew ext4'.

I do attach/upload the Outputs from Lenny for the 2 runs of e2fsck here as 'QNAP-fsck.txt'

Many thanks,
Ingo

P.S.: when my NAS is back, I probably can create such a beast of filesystem for testing ;-)

[Theodore Ts'o (tytso)]

On Wed, Aug 13, 2008 at 09:21:20PM -0000, ingo wrote:
> Many thanks Ted,
>
> So I now know where to go and I discard the unusable partitition. But I
> will get my NAS back and repaired soon (I hope) and then be able to
> create a 'virgin QNAP ext3' and check with Lenny or even Intrepid
> whether it is a ext4-alpha!

It is definitely doing something non-standard with the extents
(assuming that this wasn't a corrupted filesystem). See this:

Durchgang 1: Prüfe Inodes, Blocks, und Größen
Inode 26247171 has an ungültig extent
      (logical Block 0, ungültig physical Block 43980517704194, len 478256)
Bereinige? nein

At a guess, it's not zeroing out the high bits of the extents
structure when we extended the original Lustre extent patches to
support > 32-bit block numbers. The length looks very wrong as well.

So this is either a corrupted extent tree or an corrupted inode, or
your QNAP nas is using a very old set of ext4 extents patches.

> So for the moment, dont't worry about the ext4 story, just fsck.ext3
> should not crash when operating on such a 'homebrew ext4'.

Yep, agreed.

Can you do me a favor? Can you create a very small partition (say, 64
megabytes) on the NAS system, then pull the drive, and zero the
partition out (using dd if=/dev/zero...) then put the drive back, have
the NAS install its ersatz ext3-with-extents/early-ext4 filesystem on
it, place 2 or 3 files ranging in size from 1k to 128k, and then pull
the drive, compress the image, and send it to me? It would be very
useful for trying to figure out exactly what they are doing, and would
help reproducing the bug.

If that's too much trouble, you can just send me the results of "dd
if=/dev/sdc3 of=out.img bs=4k count=200"? That will be enough for
dumpe2fs to work, and hopefully enough for me to try reproducing the
e2fsck crash.

      - Ted

[ingo (ingo-steiner)]

Hi Ted,
many thanks for dealing with this matter! Unfortunately I cannot produce such a 'dirty' partitition/filesystem, as my QNAP-NAS heavily crashed during rebuild of that partitition sdc3 onto another disk and finally died (it is away for warranty repair ;-). The disk I have here is the disk which always remained in the NAS, I call it disk #2. Disk #1 I had pulled out and plugged in again to check whether the promised feature of hot-plugging and RAID1 really works. So, the information I can provide you with is the 'good' disk, which never was rebuilt! (I now plugged it into a external eSATA-box to collect this information).

First I copy you here the partitition table:

Platte /dev/sdc: 750.1 GByte, 750156374016 Byte
255 Köpfe, 63 Sektoren/Spuren, 91201 Zylinder
Einheiten = Zylinder von 16065 × 512 = 8225280 Bytes
Disk identifier: 0x00000000

   Gerät boot. Anfang Ende Blöcke Id System
/dev/sdc1 * 1 66 530113+ 83 Linux
/dev/sdc2 67 132 530145 82 Linux Swap / Solaris
/dev/sdc3 133 91190 731423385 83 Linux
/dev/sdc4 91191 91200 80325 83 Linux

If you now look at the size of sda3 (700GB) you understand, that I did not create an image of it before starting with fsck ;-)

Presuming that all 4 partitition have been formatted using the patched ext3, it probably helps you to have a look at the images of sda1 and/or sda4.
sda1 can be mounted, but only if you explicitely specify the option -t ext3
sda4 can be mounted, but only if you explicitely specify the option -t ext2
sda3 still crashes on Hardy e2fsck, despite Lenny's e2fsck has 'fixed' it.

What I can offer you at the moment is following:

1. partly image of current sda3 (dd if=/dev/sdc3 of=qnap-sdc3.img bs=4k count=200) -> attached!
2. bz compressed image of sdc1 (230MB)
3. bz compressed image of sdc4 (20MB)

After creating the images I again tried to e2fsck all 3 partititions under Hardy:
sdc1 and sdc4 perform correct, sdc3 crashes e2fsck.

If you are interested to have a look at QNAP's sources (should all be under GPL), please check here:
http://europe.qnap.com/download/Storage/TS-209Pro/
I guess, the sources should be in the last archive on that page (277MB).

If you need more information or upload the images of sdc1 and sdc4, please let me know,
Ingo

[ingo (ingo-steiner)]

one more information:

when trying to mount /dev/sdc3 I get following message:

mount -t ext3 /dev/sdc3 /mnt
mount: wrong fs type, bad option, bad superblock on /dev/sdc3,
       missing codepage or helper program, or other error

checking 'dmesg' afterwards tells me:

[ 7905.568588] EXT3-fs: sdc3: couldn't mount because of unsupported optional features (40).

I already googled for that message, but did not find any usefull info.
Ingo

[Theodore Ts'o (tytso)]

On Thu, Aug 14, 2008 at 11:00:44AM -0000, ingo wrote:
>
> 1. partly image of current sda3 (dd if=/dev/sdc3 of=qnap-sdc3.img bs=4k count=200) -> attached!

So I just tried downgrading to e2fsprogs 1.40.8-2ubuntu2 (and all of
its associated libraries), and I can't reproduce the crash. What I
get is:

# /sbin/e2fsck /tmp/qnap-sdc3.img
e2fsck 1.40.8 (13-Mar-2008)
/tmp/qnap-sdc3.img has unsupported feature(s): extents
e2fsck: Get a newer version of e2fsck!

That appears to be what you have installed on your system, according
to the dependencies.txt attachment. Can you reproduce the core dump
if you run "/sbin/e2fsck /tmp/qnap-sdc3.img" on your system?

                      - Ted

[Theodore Ts'o (tytso)]

On Thu, Aug 14, 2008 at 02:48:08PM -0000, ingo wrote:
> one more information:
>
> when trying to mount /dev/sdc3 I get following message:
>
> mount -t ext3 /dev/sdc3 /mnt
> mount: wrong fs type, bad option, bad superblock on /dev/sdc3,
> missing codepage or helper program, or other error
>
> checking 'dmesg' afterwards tells me:
>
> [ 7905.568588] EXT3-fs: sdc3: couldn't mount because of unsupported
> optional features (40).

This is not surprising at all; sdc3 has the extents feature, which
ext3 doesn't support. And that's where the (40) comes from:

#define EXT4_FEATURE_INCOMPAT_EXTENTS 0x0040

       - Ted

[ingo (ingo-steiner)]

Congratulations Ted!

You were right with your asumption regardind early alpha ext4. I did try to follow your recommendations on Lenny, where e2fsprogs 1.41.0 is currently standard. Issued followig commands:
-------------------
ingo@pp:~$ tune2fs -E test_fs /dev/sdc3
bash: tune2fs: command not found
ingo@pp:~$ su
Passwort:
pp:/home/ingo# tune2fs -E test_fs /dev/sdc3
tune2fs 1.41.0 (10-Jul-2008)
Setting test filesystem flag
pp:/home/ingo# mount -t ext4dev /dev/sdc3 /mnt
pp:/home/ingo#
---------------------------

Great: it mounts and I can smoothly access the data on /dev/sdc3 - see here:
---------------------------
ingo@pp:~$ ls -l /mnt
insgesamt 48
-rw------- 1 root root 6144 16. Jul 18:45 aquota.user
drwx------ 2 root root 16384 16. Jul 18:44 lost+found
drwxrwxrwx 3 root root 4096 17. Jul 22:08 Public
drwxrwxrwx 2 root root 4096 16. Jul 18:45 Qdownload
drwxrwxrwx 2 root root 4096 16. Jul 18:45 Qmultimedia
drwxrwxrwx 2 root root 4096 16. Jul 18:45 Qrecordings
drwxrwxrwx 2 root root 4096 16. Jul 18:45 Qusb
drwxrwxrwx 2 root root 4096 16. Jul 18:45 Qweb
ingo@pp:~$ ls -l /mnt/Public
insgesamt 4
drwxr-xr-x 2 500 crontab 4096 17. Jul 22:44 backup
ingo@pp:~$ ls -l /mnt/Public/backup
insgesamt 4
-rw-rw-rw- 1 ingo users 1958934821 17. Jul 22:27 sda5.tgz.000
-rw-r--r-- 1 root root 47 17. Jul 22:44 sda5.tgz.000.md5
ingo@pp:~$
-------------------------------

1e3 THANKS to YOU - all data are still there - that will be honoured by many QNAP TS-209 users!
========================================================

Now to your recent request regarding the image:
it still crashes on Hardy, see this output:

# /sbin/e2fsck ./qnap-sdc3.img
e2fsck 1.40.8 (13-Mar-2008)
./qnap-sdc3.img has unsupported feature(s):Segmentation fault (core dumped)
root@pp:/home/ingo/data/temp/QNAP-ext3+#

and I found this file in /var/crash:
50868 2008-08-14 19:45 _sbin_e2fsck.0.crash

I do attach the crash report here as well

What is amazing:
I also get crash reports from Apport in Nautilus at the same occasion, but those reports are incomplete/defect, so I cannot file a bug report for them. Appears to me that is its related or similar or even identical to the many reports on
https://bugs.launchpad.net/bugs/182345

So maybe, the origin for the crash lies in Nautilus not beheaving correctly???
Please excuse if I conclude nonsense, I am not familiar with any programming, just an attentive user and fan of Ubuntu/Debian,
Ingo

[Theodore Ts'o (tytso)]

On Thu, Aug 14, 2008 at 05:57:20PM -0000, ingo wrote:
>
> Now to your recent request regarding the image:
> it still crashes on Hardy, see this output:
>
> # /sbin/e2fsck ./qnap-sdc3.img
> e2fsck 1.40.8 (13-Mar-2008)
> ./qnap-sdc3.img has unsupported feature(s):Segmentation fault (core dumped)
> root@pp:/home/ingo/data/temp/QNAP-ext3+#

As I mentioned to you earlier, I can't reproduce the bug on my end.
The problem is the stack has been corrupted, so there's not much I can
do with the stack backtrace. Given what it has printed, it is almost
certainly in the call to e2p_feature2string --- but dumpe2fs calls
that, and without any problems on your system.

Because of where the segfault lies, it's before the newline is
printed, which localizes the code to precisely this region:

  fprintf(stderr, _("%s has unsupported feature(s):"),
   ctx->filesystem_name);

  for (i=0; i <3; i++,mask++) {
   for (j=0,m=1; j < 32; j++, m<<=1) {
    if (*mask & m)
     fprintf(stderr, " %s",
      e2p_feature2string(i, m));
   }
  }
  putc('\n', stderr);

... and e2p_feature2string doesn't call strlen() anywhere. So I'm
totally at a loss what's going on here. I've tried running valgrind
on the e2fsck 1.40.8 system, and have turned up nothing.

I guess something you can try is to see if valgrind turns up something
for you on your system. Failing that, if you are willing to build
e2fsprogs from sources, and then try to replicate the problem and run
it under gdb, we might get somewhere. Otherwise, I really don't know
what to tell you. It is a mystery what is going on here.

                           - Ted

[ingo (ingo-steiner)]

Ted,

one more thing I did not mention up to now: my system is Hardy-amd64.

My personal guess now is that it is not e2fsck itself, but in co-operation with some other part of the system (gvfs, nautilus, hal ?? whatever). And as v1.41.0 of e2fsprogs seems to handle the matter smoothly at least under Lenny, my recommendation for a fix is to just upgrade e2fsprogs and not spending more efforts in solving something which probably has been solved.

Many thanks from my side - you did me and many others a big favour by teaching how to read the 'proprietary ext3 from QNAP'. That's more than expected!

And as said, I cannot create such a virgin reproducible file system right now as my box is away for repair.
If you do not want to propose upgrade of e2fsprogs in Hardy, we will have to wait till I can create an untouched/spoiled filesystem to use for tracing.

Many thanks,
Ingo

[ingo (ingo-steiner)]

what me just came up to check the posssibility of other parts of Hardy interferring with e2fsck:

I do upload here the broken /var/cash/_usr_bin_nautilus.1000.crash
which was createt at the very same time when e2fsck crashed - does it help you further?

Ingo

[Theodore Ts'o (tytso)]

On Thu, Aug 14, 2008 at 07:30:22PM -0000, ingo wrote:
> If you do not want to propose upgrade of e2fsprogs in Hardy, we will
> have to wait till I can create an untouched/spoiled filesystem to
> use for tracing.

That's really not up to me. I'm just the upstream maintainer here.
It's up to the Ubuntu release team to decide whether they want to
upgrade to e2fsprogs 1.41, and/or a kernel that's new enough to
support ext4.

My guess is that e2fsck crashing on a filesystem which is not
supported by Ubuntu Hardy is low priority to the Ubuntu release team,
but that's not my call to make.

As far as the QNAS is concerned, if they have the code which I suspect
they do, I have a feeling that even a correctly working filesystem
from QNAS may end up generating complaints from e2fsck, but it should
*hopefully* be the case that after e2fsck fixes those corruptions so
that a modern Linux kernel is happy with it, that it will still be OK
to put that disk back in a QNAS box and it will still work correctly.
So you might want to be careful and keep lots of backups until we know
for sure that it's safe.

It is true that I'm testing on an x86 box; I don't run a 64-bit
Ubuntu, so I can't test there easily. So maybe that's why you are
seeing the problem and I am not. Unfortunately, short of running
"valgrind" or recompiling from source and running e2fsck under a
debugger with debugging symbols and without the binary being stripped,
I'm not sure I can really take this much farther.

Regards,

      - Ted

[Julien Plissonneau Duquene (julien-plissonneau-duquene)]

Marking incomplete; waiting for details with valgrind.

[Scott James Remnant (Canonical) (canonical-scott)]

The bug has expired, without the requested information.

Assuming that this is not a bug with the karmic e2fsprogs, I feel that we should close it.