squashfs kernel module does not support lzma

[Martin Pitt]

The desktop CDs currently fail to boot, since they cannot mount the squashfs.

I did some tests on a current hardy install: mounting gutsy's squashfs works fine, mounting hardy's does not. This happens with *both* 2.6.22 and 2.6.24, so the problem does not seem to be in the kernel module.

$ sudo mkdir /gutsy /hardy
$ sudo mount -o loop download/ubuntu/stable/gutsy-desktop-amd64.iso /gutsy
$ sudo mount -o loop download/ubuntu/hardy-desktop-amd64.iso /hardy

gutsy works fine:
$ sudo mount -o loop -t squashfs /gutsy/casper/filesystem.squashfs /mnt
$ mount|grep squash
/gutsy/casper/filesystem.squashfs on /mnt type squashfs (ro,loop=/dev/loop2)
$ sudo umount /mnt

hardy doesn't:
$ sudo mount -o loop -t squashfs /hardy/casper/filesystem.squashfs /mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop2,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail or so
$ dmesg | tail
[...]
[ 279.734695] squashfs: version 3.3 (2007/10/31) Phillip Lougher
[ 302.164023] SQUASHFS error: Can't find a SQUASHFS superblock on loop2

[Martin Pitt (pitti)]

Since this does not depend on the kernel driver version, I guess that squashfs-tools 0.3.3 is broken somehow?

[Steve Langasek (vorlon)]

This appears to be tied to a change in the default behavior of mksquashfs, which now uses lzma compression instead of gzip.

Since this isn't working even with 2.6.24 kernels, it may be better to revert the default compression option for the time being?

[Martin Pitt (pitti)]

Ah, we got it. squashfs-tools 3.3 switched to lzma by default, which our current kernel module does not seem to support.

We worked around the problem by calling mksquashfs with -nolzma in livecd-rootfs for now.

[Martin Pitt (pitti)]

Hm, the squashfs.ko is also version 3.3 in 2.6.24. Could that just be flipping on a kernel configuration option to add general lzma compression support?

[Amit Kucheria (amitk)]

As reported here : https://lists.ubuntu.com/archives/kernel-team/2008-March/002201.html, we are in Kernel Freeze and we can't add necessary modules since it may impact the live-cds.

[Thomas (t.c)]

any news to this topic?
LZMA Support is still missing in jaunty (9.04)

[Dominic Chen (ddcc)]

Please reopen this bug, it's still an issue in Jaunty.

[Jeremy Foshee (jeremyfoshee)]

Can you confirm this issue exists with the most recent Karmic Koala 9.10 Alpha release? ISO CD images are available at http://cdimage.ubuntu.com/releases/karmic/ . If the issue remains with Karmic it would be great to then also test the latest upstream mainline kernel available. This will allow additional upstream developers to examine this issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds .

Thanks in advance.

-JFo

[Dominic Chen (ddcc)]

I'll try, but those are pretty large iso's to download.

[Dominic Chen (ddcc)]

Fixed for me in Karmic Beta.

[Jeremy Foshee (jeremyfoshee)]

ddcc,
     Excellent news! I'll set this to Fix Released.

-JFo

[Phillip Lougher (phillip-lougher)]

No this isn't fixed, and it won't be fixed in the Karmic release timeframe. I anticipate mainlined LZMA support for Squashfs 4.0 will be available in kernel 2.6.33.

The underlying reason for this "bug" is because you got your squashfs-tools and squashfs kernel modules mixed up. Squashfs has never officially supported LZMA compression, and the official Squashfs tools and kernel module releases available from http://squashfs.org has no support for LZMA.

What complicates the situation is there are unofficial (but approved by myself) third party LZMA patches available from http://squashfs-lzma.org for Squashfs 3.x.

What appears to have happened is Debian adopted the LZMA patches for their Squashfs 3.3 packages, which was inherited by Ubuntu (which is why comment #3 reports mksquashfs 3.3 switched to LZMA by default). Debian will have patched both the squashfs-tools and the squashfs kernel code, however, while Ubuntu inherited the squashfs-tools changes, it didn't inherit the squashfs kernel code changes. This is because the Ubuntu kernel is produced independently, and the kernel code carries the original unmodified Squashfs code from http://squashfs.org. This has produced a mis-match which has existed up to and including Jaunty.

Having just tested Jaunty mksquashfs, even though it is clearly the LZMA patched mksquashfs (with the -lzma/-nolzma options), the default has changed at sometime to building gzipped squashfs filesystems which are supported by the Jaunty kernel. So the mis-match isn't as apparent as it was earlier (although the -lzma option will still produce unmountable filesystems).

Squashfs 4.0 which I mainlined in kernel 2.6.29 (and so it is the version used by Karmic) complicates the situation yet further. Squashfs 4.0 doesn't support LZMA, but there are no third-party LZMA patches available from http://squashfs-lzma.org. The site states "There is no longer any interest in developing separate patches against squashfs 4.0" as Squashfs has been mainlined. I can surmise the reason for this is first because I substantially rewrote the Squashfs 4.0 kernel code for mainlining, and so their LZMA patches won't work without a substantial rewrite, and second because the LZMA patches should preferably be mainlined themselves, and their LZMA patches aren't suitable for mainlining.

The upshot of this in the short term for Karmic is that your squashfs-tools/squashfs-kernel mismatch has gone away, because both the squashfs 4.0 tools and the kernel code does not support LZMA. However, the downside is you're not going to have LZMA support in Karmic.

In the longer term people are asking me (thanks guys) where LZMA support for Squashfs 4.0 has gone. Because Squashfs has been mainlined, and because there has also been a (poor) implementation of LZMA mainlined, my original reasons for not supporting LZMA in Squashfs no longer applies. I'm therefore currently working on getting Squashfs LZMA support mainlined. Because the kernel LZMA implementation requires substantial modifications to work efficiently with Squashfs, I don't anticipate the patches entering mainline before 2.6.33, although out of tree patches may be available before then...

[AJenbo (ajenbo)]

Sounds grate, hope to see LZMA used in Maverick Meerkat :)
Should push the CD to an equivalent of 852-926MB to the current GZIP 702MB (or 2808MB uncompressed).