Allow verifying that a snap recipe build corresponds to a store revision

[John Neffenger]

Please add the SHA-512 hash code to the build status page of a Snap package built on Launchpad. That simple change would allow users to verify that the Snap package installed on their system was in fact built on Launchpad and that the build log accurately represents its build.

For background, there's currently no way to verify that a Snap package whose build status is shown on Launchpad is the one that's actually installed on your system. That's important because there's a big difference in security between a package in the Snap Store created from closed source on a developer's workstation, on the one hand, and a package built on Launchpad from open source with a detailed build log and manifest file, on the other.

For example, my current installation of the OpenJDK Snap package contains a manifest file with the following build information:

$ cat /snap/openjdk/current/snap/manifest.yaml | grep -A3 image-info
image-info:
  build-request-id: lp-71597199
  build-request-timestamp: '2022-05-27T16:29:01Z'
  build_url: https://launchpad.net/~jgneff/openjdk-snap/+snap/openjdk-candidate/+build/1775656

Yet if I go to the build URL, there is nothing on that page that lets me verify that I am actually using the package created by Launchpad. For a brief time, the build status page contains a link to the package itself, which would allow for such verification, but that download link is deleted after a few days.

Note that Launchpad already generates the SHA-512 and SHA3-384 hash values of the package, as both codes are included on the page linked from "Manage this package in the store" on the build status. That linked page, though, is available only to the developer of the Snap, not to its users.

With this change, users could verify the source of the package with a simple:

$ sudo sha512sum /var/lib/snapd/snaps/openjdk_909.snap
3d6ccee1...c8939378 /var/lib/snapd/snaps/openjdk_909.snap

For more information on attempts to verify Snap packages built on Launchpad, see the following blog post:

How to verify the source of a Snap package
https://merlijn.sebrechts.be/blog/2020-08-17-verify-snap/

[Jürgen Gmach (jugmac00)]

Thank you for the detailed feature request.

Unfortunately, this is both not that straightforward as it seems, and also we need to involve the Snap Store team to get some feedback.

[William Grant (wgrant)]

A simple solution might be to log hashes at the end of the build log. That wouldn't get them into LP in a searchable, structured way, but it would allow someone to manually check that the .snap was produced by the build it said it was.

Snaps are often referred to by (a weird encoding of) their SHA3-384, but it might make sense to log that alongside more normal hashes too.

[Colin Watson (cjwatson)]

Just one extra note here, though Jürgen touched on it obliquely: Launchpad doesn't in fact generate the SHA-512 and SHA3-384 hashes; rather, those are generated by the store. One reason why it's not straightforward to simply display these in the Launchpad web application as well is that Launchpad no longer has any way to generate these after it's garbage-collected its local copies of the files, and it doesn't currently store them in its own database.

It might also be worth noting that you can derive the snap's name and revision from the `href` of the "Manage this package in the store" link. A malicious build can't forge that, and since revisions in the store are immutable you can use that to verify that a build corresponds to a given revision.

[John Neffenger (jgneff)]

Thank you, William.

> A simple solution might be to log hashes at the end of the build log.

Yes, a hash code in the build log would solve the problem, too! I was surprised to find no hash value at the end of the log file right after the package is created:

  Snapping...
  Snapped openjdk_18.0.1.1+2_amd64.snap
  Revoking proxy token...

I could print the hash value myself, but Snapcraft doesn't provide an override hook for the 'snap' or 'pack' step of the lifecycle.

> Snaps are often referred to by (a weird encoding of) their SHA3-384 ...

Is that the revision string?

  Build status
  [FULLYBUILT] Successfully built on lcy02-amd64-010
  Revision: a7335859baa3c35d2250d29c74ae7f2d3eb194fa

I checked that string against every hash value that 'openssl' can create, and nothing matched. What is that string?

[John Neffenger (jgneff)]

Thank you, Colin.

> A malicious build can't forge that, and since revisions in the store are immutable you can use that to verify that a build corresponds to a given revision.

So if Launchpad uploaded revision 909 of the OpenJDK Snap package, for example:

  Manage this package in the store
  <https://dashboard.snapcraft.io/snaps/openjdk/revisions/909/>

and I have revision 909 installed, they are guaranteed to be identical?

  $ snap list openjdk
  Name Version Rev Tracking Publisher Notes
  openjdk 18.0.1.1+2 909 latest/stable jgneff -

That's less satisfying than a hash value, but it still might work. Would the following definition of 'revision' convince a technical end user?

  revision
  https://snapcraft.io/docs/glossary#heading--revision

In that case, could Launchpad add the Snap package revision number to the build status page so that it's visible without having to hover over the "Manage this package" link?

[Colin Watson (cjwatson)]

The "Revision:" shown on the build index page at the moment is the git commit ID. I guess I should change that to "VCS revision:" to make it a bit less ambiguously confusing.

I've posted a merge proposal to show the store revision on the build index page.

[John Neffenger (jgneff)]

I am grateful to the the Launchpad team for taking this request seriously and responding so quickly. Thank you, Colin, for your merge proposal. I think your suggested changes solve the problem.

Technical users can find the link to the build status page in the Snap manifest file. The new information on the status page will let them verify that Launchpad built the installed Snap revision and uploaded it to the Snap Store. In addition, they will be able to use the SHA-512 hash value in the build log to verify that the Snap package mounted on their system is identical to the one built by Launchpad.

Eventually, I'd like to see us go one step further and allow for verified reproducible builds of Snap packages created by Launchpad. Users could then build a package locally and verify it as follows:

$ git clone https://github.com/jgneff/openjdk.git
$ cd openjdk
$ snapcraft
$ sha512sum openjdk_20+3_amd64.snap

Comparing the local checksum with the one logged by Launchpad would give us the ability to detect advanced supply-chain attacks against Launchpad itself, and quickly, too! For more information, see:

Preventing Supply Chain Attacks like SolarWinds
https://www.linuxfoundation.org/blog/preventing-supply-chain-attacks-like-solarwinds/

Right now, that checksum comparison fails due to just four lines in the manifest file and the timestamps in the SquashFS archive. See my comments #6 and #7 in this bug report for more information:

Bug #1890046 "Handle SOURCE_DATE_EPOCH for SquashFS"
https://bugs.launchpad.net/snapcraft/+bug/1890046

[Colin Watson (cjwatson)]

We've deployed the rework of the snap build index page.