Handle SOURCE_DATE_EPOCH for SquashFS

[Anatoli Babenia]

For reproducible snap builds is needed that if SOURCE_DATE_EPOCH is set, the timestamps inside of .snap archive were set from it.

--- v2.snap
+++ yakshaveinc_eternal_amd64.snap
├── unsquashfs -s {}
│ @@ -1,9 +1,9 @@
│ Found a valid SQUASHFS 4:0 superblock
│ -Creation or last append time Sat Aug 1 18:36:27 2020
│ +Creation or last append time Sun Aug 2 15:04:00 2020
│ Filesystem size 576 bytes (0.56 Kbytes / 0.00 Mbytes)
│ Compression xz
│ Block size 131072
│ Filesystem is exportable via NFS
│ Inodes are compressed
│ Data is compressed
│ Uids/Gids (Id table) are compressed
├── unsquashfs -d -lls {}
│ @@ -1,6 +1,6 @@
│ Parallel unsquashfs: Using 4 processors
│ 1 inodes (1 blocks) to write
│
│ drwxr-xr-x root/root 27 2020-05-26 13:34
│ drwxr-xr-x root/root 32 2020-05-26 13:34 /meta
│ --rw-r--r-- root/root 275 2020-08-01 18:36 /meta/snap.yaml
│ +-rw-r--r-- root/root 275 2020-08-02 15:04 /meta/snap.yaml

[Ian Johnson (anonymouse67)]

can you try using `unsquashfs` on the snap you get and then `snap pack`? Does `snap pack` work with the `SOURCE_DATE_EPOCH` env var?

[Anatoli Babenia (abitrolly)]

Nope, setting SOURCE_DATE_EPOCH for `snap pack` has no effect.

[Maciej Borzecki (maciek-borzecki)]

I see this happen to. I think the problem is that mksquashfs from the snapd snap (or the core snap is too old).

This is from strace when invoking `snap pack`:

[pid 57453] execve("/var/lib/snapd/snap/snapd/current/lib/x86_64-linux-gnu/ld-2.23.so", ["/var/lib/snapd/snap/snapd/current/lib/x86_64-linux-gnu/ld-2.23.so", "--library-path", "/var/lib/snapd/snap/snapd/current/usr/local/lib:/var/lib/snapd/snap/snapd/current/lib/x86_64-linux-gnu:/var/lib/snapd/snap/snapd/current/usr/lib/x86_64-linux-gnu", "/var/lib/snapd/snap/snapd/current/usr/bin
/mksquashfs", ".", "/home/maciek/work/canonical/image/snapd_2.45.3.1+git2336.g484f47d_amd64.snap", "-noappend", "-comp", "xz", "-no-fragments", "-no-progress", "-wildcards", "-ef", "/tmp/.snap-pack-exclude-981155965", "-all-root", "-no-xattrs"], ["DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus", "DISPLAY=:0", "HOME=/home/maciek", "INVOCATION_ID=6b2d474429d64baebb7b8457e0057
a7e", "JOURNAL_STREAM=8:39754", "LANG=en_US.UTF-8", "LC_TIME=pl_PL.UTF-8", "LOGNAME=maciek", "MAIL=/var/spool/mail/maciek", "MANAGERPID=1483", "PATH=/usr/local/bin:/usr/bin:/var/lib/snapd/snap/bin:/home/maciek/bin:/home/maciek/.local/bin:/usr/local/sbin:/var/lib/flatpak/exports/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl", "PLASMA_USE_QT
_SCALING=1", "PWD=/home/maciek/work/canonical/image", "SHELL=/usr/bin/zsh", "SSH_AUTH_SOCK=/run/user/1000/keyring/ssh", "TERM=screen-256color", "TMUX=/tmp//tmux-1000/default,1557,0", "TMUX_PANE=%8", "USER=maciek", "XAUTHORITY=/run/user/1000/gdm/Xauthority", "XDG_DATA_DIRS=/home/maciek/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share/:/usr/share/:/
var/lib/snapd/desktop", "XDG_RUNTIME_DIR=/run/user/1000", "SHLVL=1", "OLDPWD=/home/maciek", "KWIN_TRIPLE_BUFFER=1", "HG=/usr/bin/hg", "MOZ_PLUGIN_PATH=/usr/lib/mozilla/plugins", "EDITOR=emacsclient -t -a ''", "PAGER=less", "LC_MESSAGES=", "_=/usr/bin/strace", "SOURCE_DATE_EPOCH=1596107149"]) = 0

Clearly the environment variable is passed to mksquashfs. However, the mksquashfs binary is not aware of SOURCE_DATE_EPOCH at all:

$ strings /var/lib/snapd/snap/snapd/current/usr/bin/mksquashfs| grep SOURCE_DATE

Now my local binary is:

$ strings /usr/bin/mksquashfs| grep SOURCE_DATE
SOURCE_DATE_EPOCH
FATAL ERROR:SOURCE_DATE_EPOCH and command line options can't be used at the same time to set timestamp(s)
Env Var SOURCE_DATE_EPOCH has invalid time value

We would need to update the mksquashfs that goes into the snap binary for this to work. The version we currently have comes from 16.04.

[Anatoli Babenia (abitrolly)]

Need at least version 4.4

https://lore.kernel<email address hidden>/

[Jamie Strandboge (jdstrand)]

The squashfs-tools in xenial are patched to support -fstime which I think could be used to achieve the same result. Eg, snapd, if SOURCE_DATE_EPOCH is set and verified, add '-fstime $SOURCE_DATE_EPOCH' to the mksquashfs arguments.

[John Neffenger (jgneff)]

The build of the OpenJDK Snap package is entirely reproducible except for the timestamps identified by this bug report and the 'snap/manifest.yaml' file added by Launchpad.

I'll attach the file 'openjdk.html' showing the difference in the timestamps in the "SQUASHFS 4:0 superblock" and the timestamps of some of the files. It was created by 'diffoscope' when comparing two locally-built OpenJDK Snap packages.

The only other difference is in the manifest file added by Launchpad when building remotely:

==================================================
$ diff -r --no-dereference remote1/openjdk_19+21_amd64/snap/manifest.yaml \
    remote2/openjdk_19+21_amd64/snap/manifest.yaml
2c2
< snapcraft-started-at: '2022-05-05T20:33:55.570239Z'
---
> snapcraft-started-at: '2022-05-05T22:24:38.683744Z'
865,867c865,867
< build-request-id: lp-71105162
< build-request-timestamp: '2022-05-05T20:22:55Z'
< build_url: https://launchpad.net/~jgneff/+snap/snapcraft-openjdk-4f34018c466452df809721dd095d9a4e/+build/1753738
---
> build-request-id: lp-71106341
> build-request-timestamp: '2022-05-05T21:51:34Z'
> build_url: https://launchpad.net/~jgneff/+snap/snapcraft-openjdk-4a22e98de08b515d53b26d9d98a7e7d4/+build/1753760
==================================================

To fully support reproducible builds even for snaps built on Launchpad, it seems we'll need some way to pass in the 'SOURCE_DATE_EPOCH' variable to the Launchpad build environment, or let Launchpad pick some fixed instant. The 'jdk' part of my Snap package adds the environment variable in the 'override-build' scriptlet, but I don't see any way to set it for the 'snap pack' command. See:

https://github.com/jgneff/openjdk/blob/edge/snap/snapcraft.yaml#L123

I realize we'll likely need a separate Launchpad bug report, but I just thought I might show how close we are to having fully reproducible builds. :-)

[John Neffenger (jgneff)]

File created with:

$ diffoscope --html openjdk.html openjdk_19+21_amd64-?.snap

[Colin Watson (cjwatson)]

In general I'm in favour of efforts to ensure reproducible builds, but I'm concerned about interactions with the store. The store rejects attempts to upload blobs that are the same as ones it's seen before. At the moment it's possible to work around certain kinds of upload failures just by triggering a new build, taking advantage of the lack of reproducibility. Has anyone talked to the store team about whether it might be possible to relax this restriction somehow?

It also seems specifically undesirable to lose things like `build_url`, and `build-request-id` and build-request-timestamp` are used by the build frontend on snapcraft.io.

[Anatoli Babenia (abitrolly)]

> At the moment it's possible to work around certain kinds of upload failures just by triggering a new build, taking advantage of the lack of reproducibility.

It doesn't look like workarounds for store bugs is a good reason to avoid having reproducible builds. If there is an upload failure that doesn't depend on file content, the store should reject the upload without blocking retries. But there is also a question, why uploads are failing at all? Can you reference the specific issue on the store site?

[Colin Watson (cjwatson)]

It's possible I never actually filed it, though I've had brief hallway conversations with store folks about it in the past. I'll make sure to file it the next time I run across it - in the past that's been while looking for large snaps in Launchpad that weren't successfully uploaded to the store, in the process of trying to trim space in the Launchpad librarian.

The more serious issue is that we currently embed information about the build in the snap. I'm not willing to simply disable that, because it's the closest thing we have to an audit trail at the moment for where Launchpad-built snaps were built. It might be possible to come up with another way to store that information (perhaps an assertion, though build-provided assertions were a bit of a can of worms last I checked), but it would need some careful planning.

[Anatoli Babenia (abitrolly)]

Why not to make reproducible builds opt-in for the start? So that if SOURCE_DATE_EPOCH, then make reproducible build. Launchpad won't have SOURCE_DATE_EPOCH enabled until you're ready to handle that. Win win, no?