blkid library detects JFS partition as FAT32

[boga]

blkid detects a JFS partition as FAT32. Thus fsck fails on this partition unless the file system is specified explicitly or by device name (rather than UUID) in fstab. This causes automatic fsck on startup failures resulting in boot failures.
E.g.:
xxx@yyy:~$ sudo blkid -c /dev/null
/dev/sda1: UUID="d7e37603-b7ee-4994-add8-db53b71e6dd5" TYPE="jfs"
/dev/sdb1: SEC_TYPE="msdos" LABEL="DOS" UUID="E7D3-A415" TYPE="vfat"
/dev/sdb3: UUID="42d7432e-3d38-4ac4-ae08-706c8bf3e64f" TYPE="ext3"
/dev/sdb5: SEC_TYPE="msdos" UUID="5593-65BD" TYPE="vfat"
/dev/sdb6: TYPE="swap" UUID="629513d4-df93-48c5-be48-2e7314262747"
/dev/sdc1: SEC_TYPE="msdos" UUID="479E-4CF4" TYPE="vfat"

/dev/sdb5 is a JFS partition indeed. Strangely /dev/sda1 which is also a JFS partition is detected correctly. However the difference is that /dev/sda1 is formatted without -O option (case sensitive) whereas /dev/sda5 is formatted with -O option (case insensitive)

[boga (boga-inbox)]

Studying lib/blkid/probe.c: the problem occurs because the JFS partition matches the type_array[] entry:
  { "vfat", 0, 0x1fe, 2, "\125\252", probe_fat_nomagic },
The problem can be fixed by moving JFS-related entry before this VFAT entry.

[Theodore Ts'o (tytso)]

Can you send me the first 64k of the device? i.e.:

sudo dd if=/dev/sdb5 of=/tmp/jfs.img bs=64k count=1

And then upload /tmp/jfs.img

I have no doubt that moving the JFS-related entry before the VFAT entry will solve the problem, but the probe_fat_nomagic() does a lot of tests before concluding that the filesystem is VFAT. This leads me to believe that /dev/sdb5 was previously formatted as VFAT, and the JFS mkfs didn't do enough to eradicate the traces of the old VFAT partition. The problem is that there is a much weaker test for JFS, so this could lead to false positives in the other direction. (i.e., a JFS filesystem which is then formatted to be a VFAT partition.) So I'd like to get a copy of the beginning of the filesystem to see if there's a better fix.

My guess is the other problem is that mkfs.jfs needs to do a better job wiping out the first few kilobytes of the image to prevent this kind of false positive.

[boga (boga-inbox)]

I've attached it to the e-mail listed in your profile.

[boga (boga-inbox)]

May be you can improve JFS test strength by adding check for the correct relations between the superblock fields: s_bsize, s_l2bsize, s_l2bfactor, s_pbsize, s_l2pbsize?

[Theodore Ts'o (tytso)]

(Sorry for the delay in getting back to you; I've been travelling.)

Hmm, looking at what you sent me, it looks like DFSee was being
"clever". It looks like it deliberately put something which looks
like almost exactly like a FAT12/FAT16 header, passing all of the
normal checks one might make for seeing if the various fields make
sense.

(gdb) p *((struct msdos_super_block *) buf)
$14 = {ms_ignored = "�\037", ms_sysid = "DFSee8.x",
  ms_sector_size = "\000\002", ms_cluster_size = 8 '\b', ms_reserved = 1,
  ms_fats = 2 '\002', ms_dir_entries = "\000\002", ms_sectors = "\000",
  ms_media = 248 '�', ms_fat_length = 134, ms_secs_track = 63, ms_heads = 255,
  ms_hidden = 63, ms_total_sect = 16868184, ms_unknown = "\200\001)",
  ms_serno = "�e\223U", ms_label = "\000\000\000\000\000\000\000\000\000\000",
  ms_magic = "JFS ",
  ms_dummy2 = " y\027\000\000@\00097\000\fy\027\000\r\000 9O\000d\000\b\000\b\000px\002\000S\000\000\000�w\002\034\000\000\000\000�w\002\034\200x\002\000�����}\002\034\000\000\000\000^\003\000\000�\205\000\000�x\002\000����\b\000\000\000\002\000\000\000\000\000\000'|x\002\000(y\002\000�\036�\037\b\000\000\000\b\000\000\000d\000\000\000 9\t\000\r\000\000\000\fy\002\000\0009\006\000\000@\000\000 y\002\000(y\002\000\000\000\000\000\000\000\000\000\200y\002\000\000@\000\000 \000\000\000�y$\000\033\000\004\000\000\000\000\000\0009\006\000�y\002\000�y\210x\002\034�y\027", ms_pmagic = "\000\026"}

The reason why I don't just check ms_magic is that there have been
some (badly behaved) FAT filesystem formatters which don't set
ms_magic, or set it to some garbage value other than "FAT12 " or
"FAT16 ".

What I suspect I will end up doing is checking to see if ms_sysid
begins with "DFSee" (since I assume "8.x" is the version number), and
if it is set, I will insist on ms_magic being "FAT12 " or "FAT16 ".

Can you do me a favor? Can you create for me a series of filesystems
that are FAT12, FAT16, FAT32, and perhaps a couple of other
filesystems that DFSee supports, using your version of DFSee, and send
me the first 32k of said filesystems? I want to see how what sort
"magic" it is putting into those filesystems, so I can arm blkid
appropriately to detect them.

I've also cc'ed the author of DFSee in hopes that he might be able to
shed a bit more light on what's going on here.

      - Ted

[Theodore Ts'o (tytso)]

FYI, this is what I've checked into the e2fsprogs git repository.

                                    - Ted

commit 8305cdf401b05e3748b87d0b63c4fa5bad8f8eb6
Author: Theodore Ts'o <email address hidden>
Date: Sat Aug 23 22:11:01 2008 -0400

    libblkid: Fix false detection of DFSee created filesystems.

    DFSee appears to create a pseudo FAT-12/16 header in the first 512
    bytes of a filesystem which looks enough like a FAT-12/16 to fool
    blkid. Part of this is because we don't require ms_magic or vs_magic
    to be the strings "FAT12 ", "FAT16 ", or "FAT32 ", since some FAT
    filesystem formatters don't set ms_magic or vs_magic. To address
    this, we add an explicit test for DFSee in the ms_sysid field, and if
    ms_sysid has a DFSee signature, then we require the ms_magic/vs_magic
    fields to be filled in appropriately.

    Signed-off-by: "Theodore Ts'o" <email address hidden>

diff --git a/lib/blkid/probe.c b/lib/blkid/probe.c
index e2d8a25..31f3c74 100644
--- a/lib/blkid/probe.c
+++ b/lib/blkid/probe.c
@@ -587,6 +587,20 @@ static int probe_fat_nomagic(struct blkid_probe *probe,
  if (!vs->vs_fats)
   return 1;

+ /*
+ * The DFSee product apparently creates a FAT12-like boot
+ * block for non-FAT filesystems. It seems to do this at
+ * least for JFS (see Launchpad Bug #255255) and possibly
+ * others. We already check for the FAT12/16/32 magic strings
+ * in the probe table, so if we got here, it must be because
+ * we have something created by DFSee but which isn't actually
+ * a FAT filesystem. I'm assuming here that DFSee actually
+ * does fill in ms_magic and vs_magic correctly, but that
+ * seems like a safe bet.
+ */
+ if (memcmp(vs->vs_sysid, "DFSee", 5) == 0)
+ return 1;
+
  return probe_fat(probe, id, buf);
 }

[Theodore Ts'o (tytso)]

On Wed, Aug 06, 2008 at 03:34:30PM -0000, boga wrote:
> May be you can improve JFS test strength by adding check for the correct
> relations between the superblock fields: s_bsize, s_l2bsize,
> s_l2bfactor, s_pbsize, s_l2pbsize?

This is also a good idea, even if I don't move the probe order of the
tests around. (The patch to explicitly check for DFSee should solve
your particular issue.)

      - Ted

commit ddd46b9b85c158fb51c0ba45ec11b7c032fb457f
Author: Theodore Ts'o <email address hidden>
Date: Sat Aug 23 22:27:22 2008 -0400

    libblkid: Strength the JFS probe routine

    Check to make sure a JFS filesystem is really correct by checking the
    relationship between the following fields in the JFS superblock:
    s_bsize, s_l2bsize, s_pbsize, s_l2pbsize, and s_l2bfactor. Thanks to
    Lesh Bogdanow for this suggestion.

    Signed-off-by: "Theodore Ts'o" <email address hidden>

diff --git a/lib/blkid/probe.c b/lib/blkid/probe.c
index 31f3c74..91db9a1 100644
--- a/lib/blkid/probe.c
+++ b/lib/blkid/probe.c
@@ -781,6 +781,16 @@ static int probe_jfs(struct blkid_probe *probe,

  js = (struct jfs_super_block *)buf;

+ if (blkid_le32(js->js_bsize) != (1 << blkid_le16(js->js_l2bsize)))
+ return 1;
+
+ if (blkid_le32(js->js_pbsize) != (1 << blkid_le16(js->js_l2pbsize)))
+ return 1;
+
+ if ((blkid_le16(js->js_l2bsize) - blkid_le16(js->js_l2pbsize)) !=
+ blkid_le16(js->js_l2bfactor))
+ return 1;
+
  if (strlen((char *) js->js_label))
   label = (char *) js->js_label;
  blkid_set_tag(probe->dev, "LABEL", label, sizeof(js->js_label));
diff --git a/lib/blkid/probe.h b/lib/blkid/probe.h
index ac3a083..993156d 100644
--- a/lib/blkid/probe.h
+++ b/lib/blkid/probe.h
@@ -181,10 +181,13 @@ struct jfs_super_block {
  unsigned char js_magic[4];
  __u32 js_version;
  __u64 js_size;
- __u32 js_bsize;
- __u32 js_dummy1;
- __u32 js_pbsize;
- __u32 js_dummy2[27];
+ __u32 js_bsize; /* 4: aggregate block size in bytes */
+ __u16 js_l2bsize; /* 2: log2 of s_bsize */
+ __u16 js_l2bfactor; /* 2: log2(s_bsize/hardware block size) */
+ __u32 js_pbsize; /* 4: hardware/LVM block size in bytes */
+ __u16 js_l2pbsize; /* 2: log2 of s_pbsize */
+ __u16 js_pad; /* 2: padding necessary for alignment */
+ __u32 js_dummy2[26];
  unsigned char js_uuid[16];
  unsigned char js_label[16];
  unsigned char js_loguuid[16];

[Theodore Ts'o (tytso)]

On Sun, Aug 24, 2008 at 11:28:42AM +0100, Jan van Wijk wrote:
> >Hmm, looking at what you sent me, it looks like DFSee was being
> >"clever". It looks like it deliberately put something which looks
> >like almost exactly like a FAT12/FAT16 header, passing all of the
> >normal checks one might make for seeing if the various fields make
> >sense.
>
> Those fields contents (similar to FAT) are actually standard for
> any JFS filesystem formatted by the OS/2 FORMAT command
> (or rather UJFS.DLL on an OS/2 or eComStation system :-)

Thanks, that's useful information. The filesystem in question was
almost certainly created by a Linux user who didn't use OS/2 to format
his partition --- and /sbin/mkfs.jfs zeros out the 32k of the
filesystem to avoid confusion by filesystem identification programs
like blkid, vol_id, disktype, et. al.

And while I do care about making blkid be able to support ancient
filesystems for obsolete operating systems (and the
file-system-sampler from disktype.cvs.sourceforge.net has an excellent
set of filesystem exemplars for testing, including an Atari ST floppy
and a BeOS BFS filesystem), the primary goal is to be able to detect
filesystems that are likely to be used by Linux users --- so I've only
seen this situation with this pseudo FAT12/16 header with DFSee.
Maybe there are other partition management programs that cause this,
but I've not seen it in my experience.

> >The reason why I don't just check ms_magic is that there have been
> >some (badly behaved) FAT filesystem formatters which don't set
> >ms_magic, or set it to some garbage value other than "FAT12 " or
> >"FAT16 ".
>
> OK, understand, but you may see the same 'FAT like fields' for OS/2
> formatted JFS filesystems with a ms_sysid of 'IBM 4.50' ...
>
> Actually, bootsectors for an HPFS filesystems look very much like
> these as well, so you may see a similar behaviour there too.

Hmm, so maybe what I should do is put in an explicit check for "JFS "
and "HPFS " in the ms_magic field (at offset 0x36) and fail it as a
FAT filesystem if I see those magic values there. That should cover
the situation with OS/2 formatted filesystems.

Out of curiosity, does DFSee try to do something similar with any
other filesystems? Or is this something you only did for JFS and HPFS
to replicate what OS/2 was doing?

> I assume your code (that analyses the bootsector) does take into
> account that the 'ms_magic' field on FAT32 is at a different offset ?
> It is at 0x36 for FAT16 (and others) but at 0x52 for FAT32

Oh yes, e2fsprogs definitely checks for "FAT32 " and "MSWIN" at offset
0x52.

One of the things which confused me is why DFSee (and OS/2,
apparently, according to your reports) is using a FAT12/16 style
pseudo-superblock, and not a FAT32 style superblock. I also wonder
what programs were actually looking at the FAT12/16-style pseudo
superblock information for JFS and HPFS filesystems, since clearly
Linux doesn't need it, and as far as I know the MBR boot sector
doesn't require it to be present.

In any case, thanks, Jan, for your comments, they have been very
helpful!

      - Ted

P.S. Here's the new version of the patch which I've applied to blkid.

commit ...

[boga (boga-inbox)]

Theodore,
DFSee seems to have originated as an OS/2 partition management utility so there's no wonder it simulates OS/2 UHPFS.DLL/UJFS.DLL behaviour. I've made some tests and the problem seems to involve only those file systems (e.g. when creating an ext2/ext3/reiser partition DFSee simply fills the boot sector with F6).
The reason for using FAT12/16 style pseudo-superblock seems to be quite simple: there was no FAT32 at the time HPFS was developed.
So I guess your latest idea is correct: if a boot sector looks like FAT12/16 and contains "JFS " or "HPFS " at 0x36 then it is not FAT. However are you sure about memcmp(ms->ms_magic, "JFS ", 8) and memcmp(ms->ms_magic, "HPFS ", 8) ? They seem not to have enough spaces.

[Theodore Ts'o (tytso)]

On Mon, Aug 25, 2008 at 11:22:43PM -0000, boga wrote:

> So I guess your latest idea is correct: if a boot sector looks like
> FAT12/16 and contains "JFS " or "HPFS " at 0x36 then it is not
> FAT. However are you sure about memcmp(ms->ms_magic, "JFS ", 8) and
> memcmp(ms->ms_magic, "HPFS ", 8) ? They seem not to have enough
> spaces.

What I checked in has enoguh spaces. I think that got zapped by the
my e-mail MUA auto-fill function, and I didn't catch it.

                            - Ted

[Theodore Ts'o (tytso)]

Note: a fix for this bug was released in e2fsprogs 1.41.1.

[Julien Plissonneau Duquene (julien-plissonneau-duquene)]

Fix released in 1.41.2-1ubuntu1.