Backport support for the -lln command-line option to unsquashfs for 18.04

[Alex Murray]

[Impact]
unsquashfs gains a new flag to list numeric uid/gid in the contents of squashfs files. This can be used by the snap store dashboard and review-tools to accurately assess the owner of files within a snap etc.

[Test Case]
snap download core
unsquashfs -lln core_*.snap

The above should succeed and output numeric uid/gid information, e.g.:

...
drwxrwsr-x 0/50 3 2016-04-13 05:44 squashfs-root/var/local
drwxrwxr-x 0/114 94 2021-04-28 17:53 squashfs-root/var/log
-rw------- 0/43 0 2021-04-28 17:49 squashfs-root/var/log/btmp
-rw-r----- 0/4 31 2021-04-28 17:49 squashfs-root/var/log/dmesg
-rw-r----- 0/4 31 2021-04-28 17:49 squashfs-root/var/log/fsck/checkfs
-rw-r----- 0/4 31 2021-04-28 17:49 squashfs-root/var/log/fsck/checkroot
-rw-rw-r-- 0/43 30660 2021-04-28 17:49 squashfs-root/var/log/lastlog
-rw-rw-r-- 0/43 0 2021-04-28 17:49 squashfs-root/var/log/wtmp
drwxrwsr-x 0/8 3 2021-04-28 17:49 squashfs-root/var/mail
drwx-wx--T 0/102 3 2016-04-06 07:29 squashfs-root/var/spool/cron/crontabs
drwx------ 108/4 3 2019-03-25 23:44 squashfs-root/var/spool/rsyslog
...

As a more detailed test, the test-squashfs-tools.py script from qa-regression-testing can be used to exercise both this -lln output option as well as the existing -lls output option since https://git.launchpad.net/qa-regression-testing/commit/?id=2bc14c225b91b7283e57e2d63a88c164be43df5a to ensure both function as expected.

[Where problems could occur]
unsquashfs could fail to properly run and so the snap store review-tools could fail as a result. The review-tools and the store are coded to fail-closed such that any resulting problems with unsquashfs should not allow snaps that would otherwise be flagged by the review-tools to pass automated review.

------- original description follows --------

The snap store review-tools is changing to compare snaps based on their numeric uid/gids rather than user/group names (see https://code.launchpad.net/~alexmurray/review-tools/+git/review-tools/+merge/403103 for more details).

Currently the snap store dashboard, where the review-tools are run, is running Ubuntu 18.04 - which is based on squashfs-tools 4.3 - however support for the -lln command-line option was only added in the 4.4 release. This change https://github.com/plougher/squashfs-tools/commit/a78a589a55a8524ef31ba06ea7c752b8aa8b3604 is relatively simple and can be backported to squashfs-tools in 18.04 with minimal effort.

[Alex Murray (alexmurray)]

This update is being prepared in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4560

[Alex Murray (alexmurray)]

Autopkgtests all pass: https://bileto.ubuntu.com/excuses/4560/bionic.html

[Chris Halse Rogers (raof)]

Hello Alex, or anyone else affected,

Accepted squashfs-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/squashfs-tools/1:4.3-6ubuntu0.18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Alex Murray (alexmurray)]

I tested the update in -proposed as follows:

# enable -proposed in a fresh 18.04 LTS install
cat <<EOF | sudo tee /etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF

cat <<EOF | sudo tee /etc/apt/preferences.d/proposed-updates
# Configure apt to allow selective installs of packages from proposed
Package: *
Pin: release a=$(lsb_release -cs)-proposed
Pin-Priority: 400
EOF

# get the test script from qa-regression-testing
sudo apt install git python-minimal
git clone https://git.launchpad.net/qa-regression-testing

# package the test script etc
cd qa-regression-testing/
./scripts/make-test-tarball ./scripts/test-squashfs-tools.py

# run the test with the current unpatched version
cd /tmp
tar xf qrt-test-squashfs-tools.tar.gz
cd qrt-test-squashfs-tools/
sudo ./test-squashfs-tools.py
Skipping private tests
F.
======================================================================
FAIL: test_mksquashfs_unsquashfs (__main__.SquashfsToolsTest)
Test mksquashfs/unsquashfs - root
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test-squashfs-tools.py", line 254, in test_mksquashfs_unsquashfs
    self._verify_unsquashfs(numeric_uid, fn, listing)
  File "./test-squashfs-tools.py", line 218, in _verify_unsquashfs
    (raw, entries) = self._get_comparable_unsquashfs(numeric_uid, fn)
  File "./test-squashfs-tools.py", line 171, in _get_comparable_unsquashfs
    "unsquashfs " + switch + " failed: %s" % report)
AssertionError: unsquashfs -lln failed: SYNTAX: unsquashfs [options] filesystem [directories or files to extract]
 -v[ersion] print version, licence and copyright information
 -d[est] <pathname> unsquash to <pathname>, default "squashfs-root"
 -n[o-progress] don't display the progress bar
 -no[-xattrs] don't extract xattrs in file system
 -x[attrs] extract xattrs in file system (default)
 -u[ser-xattrs] only extract user xattrs in file system.
    Enables extracting xattrs
 -p[rocessors] <number> use <number> processors. By default will use
    number of processors available
 -i[nfo] print files as they are unsquashed
 -li[nfo] print files as they are unsquashed with file
    attributes (like ls -l output)
 -l[s] list filesystem, but don't unsquash
 -ll[s] list filesystem with file attributes (like
    ls -l output), but don't unsquash
 -f[orce] if file already exists then overwrite
 -s[tat] display filesystem superblock information
 -fstime display filesystem superblock time
 -e[f] <extract file> list of directories or files to extract.
    One per line
 -da[ta-queue] <size> Set data queue to <size> Mbytes. Default 256
    Mbytes
 -fr[ag-queue] <size> Set fragment queue to <size> Mbytes. Default
    256 Mbytes
 -r[egex] treat extract names as POSIX regular expressions
    rather than use the default shell wildcard
    expansion (globbing)

Decompressors available:
 gzip
 lzma
 lzo
 lz4
 xz

----------------------------------------------------------------------
Ran 2 tests in 2.152s

FAILED (failures=1)

# update...

[Launchpad Janitor (janitor)]

This bug was fixed in the package squashfs-tools - 1:4.3-6ubuntu0.18.04.2

---------------
squashfs-tools (1:4.3-6ubuntu0.18.04.2) bionic; urgency=medium

  * debian/patches/0012-unsquashfs-add-new-lln-option.patch,
    debian/manpages/unsquashfs.1: add support for numeric uid/gid output
    via -lln command-line option (LP: #1929370).

 -- Alex Murray <email address hidden> Fri, 21 May 2021 17:36:24 +0930

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for squashfs-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.