Make squashfs-tools in Xenial in sync with Bionic and Cosmic

[Stéphane Graber]

squashfs-tools upstream hasn't changed in a while but a number of bugfixes are applied through packaging in Debian and Ubuntu.

The bionic and cosmic versions right now are identical but xenial is missing a few fixes which is a problem for the LXD snap among other things.

Looking at debian/series/patches, the fixes currently missing in the xenial version are:
 - 0007-fix-2GB-limit-in-mksquashfs.patch
 - 0008-preserve_file_capabilities.patch

I'll attach test cases for both of those below and then will prepare an SRU that effectively makes the source package identical to what we have in bionic, minus the different changelog.

This should be pretty safe considering both Ubuntu and Debian have been shipping those two patches for a while and the fs caps one is going to be pretty important moving forward as we're discussing having Ubuntu ship with fscaps by default.

[Stéphane Graber (stgraber)]

== preserve file capabilities ==
=== Rationale ===
Filesystem capabilities aren't properly restored during unsquashfs, this effectively prevents any LXD image from containing file capabilities and breaks basic tools like mtr in recent images.

=== Testcase ===
For the fscaps part, easiest is to grab the latest cosmic cloud image from:
  http://cloud-images.ubuntu.com/cosmic/current/cosmic-server-cloudimg-amd64.squashfs

Then unsquashfs as root and check if "/usr/bin/mtr-packet" shows a capability when running "getcap".

=== Regression potential ===
Minimal, we're syncing to the exact same source and patches as we've got in bionic and cosmic, the patch is also trivial and "obviously right". The issue is that the kernel will strip capabilities during chown() and unsquashfs was restoring capabilities before calling chown. This patch simply re-orders it so that capabilities are applied after ownership.

[Stéphane Graber (stgraber)]

== fix 2GB limit in mksquashfs ==
=== Rationale ===
I don't have a direct use for this patch, but it seemed simpler and easier to just sync the entire stack of bugfixes with what we have in bionic and cosmic, so including this one too.

This fixes the case where a user attempts to append to a squashfs larger that's larger than 2GB.

=== Test case ===
Generate a large squashfs, easiest to achieve it is to turn off most compression options and feed it a large random file. Size should exceed 2GB.

Run mksquashfs with some random extra data to append and the pre-generated big squashfs.

Prior to this fix, you should get a failure like "Lseek on destination failed: Invalid argument".

With the fix applied, mksquashfs should just append the data and the resulting squashfs should be mountable and contain the new data.

=== Regression potential ===
This is another very small patch that looks "obviously correct" in that squashfs was simply overflowing an int in this case. The fix switches the type to "off_t" avoiding this issue.

The same fix has been in Ubuntu and Debian for a while so has seen a good amount of exposure.

[Adam Conrad (adconrad)]

Hello Stéphane, or anyone else affected,

Accepted squashfs-tools into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/squashfs-tools/1:4.3-3ubuntu2.16.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Stéphane Graber (stgraber)]

I confirmed that I can create a squashfs larger than 2GB (used 2.1GB), then append a file to it (used a 50MB file) and the resulting squashfs can be unpacked just fine.

```
root@xenial-32:~# dd if=/dev/urandom of=big bs=10M count=210
210+0 records in
210+0 records out
2202009600 bytes (2.2 GB, 2.1 GiB) copied, 147.973 s, 14.9 MB/s
root@xenial-32:~# mksquashfs big big.squashfs
Parallel mksquashfs: Using 8 processors
Creating 4.0 filesystem on big.squashfs, block size 131072.
[==========================================================================================================================================/] 16800/16800 100%

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
 compressed data, compressed metadata, compressed fragments, compressed xattrs
 duplicates are removed
Filesystem size 2150400.52 Kbytes (2100.00 Mbytes)
 100.00% of uncompressed filesystem size (2150465.88 Kbytes)
Inode table size 353 bytes (0.34 Kbytes)
 0.52% of uncompressed inode table size (67282 bytes)
Directory table size 21 bytes (0.02 Kbytes)
 84.00% of uncompressed directory table size (25 bytes)
Number of duplicate files found 0
Number of inodes 2
Number of files 1
Number of fragments 1
Number of symbolic links 0
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 1
Number of ids (unique uids + gids) 1
Number of uids 1
 root (0)
Number of gids 1
 root (0)
root@xenial-32:~# dd if=/dev/urandom of=small bs=10M count=5
5+0 records in
5+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 3.5235 s, 14.9 MB/s
root@xenial-32:~# mksquashfs small big.squashfs
Found a valid exportable SQUASHFS superblock on big.squashfs.
 Compression used gzip
 Inodes are compressed
 Data is compressed
 Fragments are compressed
 Xattrs are compressed
 Fragments are present in the filesystem
 Always-use-fragments option is not specified
 Duplicates are removed
 Xattrs are stored
 Filesystem size 2150400.52 Kbytes (2100.00 Mbytes)
 Block size 131072
 Number of fragments 1
 Number of inodes 2
 Number of ids 1

Parallel mksquashfs: Using 8 processors
Scanning existing filesystem...
Read existing filesystem, 1 inodes scanned
Appending to existing 4.0 filesystem on big.squashfs, block size 131072
All -b, -noI, -noD, -noF, -noX, no-duplicates, no-fragments, -always-use-fragments,
-exportable and -comp options ignored

If appending is not wanted, please re-run with -noappend specified!

Recovery file "squashfs_recovery_big.squashfs_256" written
If Mksquashfs aborts abnormally (i.e. power failure), run
mksquashfs dummy big.squashfs -recover squashfs_recovery_big.squashfs_256
to restore filesystem

[==============================================================================================================================================-] 400/400 100%

Exportable Squashfs 4.0 filesystem, gzip compressed, data block size 131072
 compressed data, compressed metadata, compressed fragments, compressed xattrs
 duplicates are removed
Filesystem size 2201600.58 Kbytes (2150.00 Mbytes)
 100.00% of uncompressed filesystem size (2201667.49 Kbytes)
Inode table size 388 bytes (0.38 Kbytes)
 0.56% of uncompressed inode table size (...

[Stéphane Graber (stgraber)]

For the fscaps part, I've created a simple file, made it cap_net_raw+pe and made a squashfs out of it, prior to this update, unpacking it showed me:
root@djanet:~/a# getcap squashfs-root/blah
root@djanet:~/a#

After the upgrade, unsquashfs with the same squashfs file gets me:
root@djanet:~/a# getcap squashfs-root/blah
squashfs-root/blah = cap_net_raw+ep
root@djanet:~/a#

[Stéphane Graber (stgraber)]

Marked as verified.

[Launchpad Janitor (janitor)]

This bug was fixed in the package squashfs-tools - 1:4.3-3ubuntu2.16.04.3

---------------
squashfs-tools (1:4.3-3ubuntu2.16.04.3) xenial; urgency=medium

  * Re-number current patches to match bionic/cosmic:
    - 0001-kfreebsd.patch
    - 0002-fix_phys_mem_calculation.patch
    - 0003-CVE-2015-4645_and_CVE-2015-4646.patch
    - 0004-unsquashfs-add-support-for-LZMA-magics.patch
    - 0005-add-fstime.patch
    - 0006-uptream-fix-race.patch
    - 0009-unsquashfs-preserve-symlink-times.patch
    - 0010-use-macros-not-raw-octal-with-chmod.patch
    - 0011-also-set-stickybit-as-non-root.patch

  * Sync patch content with bionic/cosmic (fuzz).
    - 0005-add-fstime.patch: Fix -Wint-conversion warning by
      initializing the time_t variable with (time_t)0 instead of NULL
    - 0006-uptream-fix-race.patch: Fix typo in description
    - 0003-CVE-2015-4645_and_CVE-2015-4646.patch: Fix typo in description

  * Cherry-pick two new fixes from bionic/cosmic: (LP: #1785499)
    - 0007-fix-2GB-limit-in-mksquashfs.patch
    - 0008-preserve_file_capabilities.patch

 -- Stéphane Graber <email address hidden> Sun, 05 Aug 2018 23:49:09 -0400

[Stéphane Graber (stgraber)]

The verification of the Stable Release Update for squashfs-tools has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.