Some SPDX 3.1 license identifiers are unknown to snapd

[Daniel Manrique]

In the process of updating the snap store's set of SPDX license identifiers as filed in this bug:

https://bugs.launchpad.net/snapstore/+bug/1781013

I happened upon snapd's set of known identifiers:

https://github.com/snapcore/snapd/blob/master/spdx/licenses.go

Some new identifiers (example, LGPL-2.0-only) are not known to snapd, it might be a good idea to update this list.

The list of SPDX license identifiers is here:

https://spdx.org/licenses/

This is updated constantly: for example, version 3.2 was just published yesterday.

Version: 3.2 2018-07-10

Luckily, and unlike the previous version, they now provide machine-readable files, so it should be possible to fetch a recently-updated json list of licenses and process it into something snapd will understand.

https://github.com/spdx/license-list-data

Specifically:

https://github.com/spdx/license-list-data/tree/master/json

[Daniel Manrique (roadmr)]

A proposal we came up with:

- The store will use the latest version of the SPDX license list from the location noted above. We will update our version on every store rollout (happens several times a week).
- Since snapds in the wild are not necessarily always in sync and up to date, there is always the possibility snapd will receive from the store a license expression with unknown (read: new) identifiers. Even having the store use snapd as the validation engine would not remove this possibility.
- So snapd could get/refresh the list of known identifiers from the store. The store will have a verbatim copy of the .json files from spdx.
- To avoid excess traffic, Bret suggested:
  1- when trying to validate a snap's license, use the local data
  2- If an unknown identifier is found in an expression, try fetching the latest data from the store, and retry the validation (which should now pass). Cache that latest data to keep the local license list updated.
  3- If the validation still fails, then it is a bogus expression; show the appropriate error.

We also need to consider the sideloading case (e.g. snapd could maybe have a cached list in the event of sideload or whatever). A problem with sideloads is that if a snap with a newish license expression is installed and snapd has no store access, it will be unable to update its expression. In this case I would suggest just saying it is an unknown license to this snapd.

In any case, snapd should have an initial, seeded list of licenses which should be updated periodically so the disconnected and unfrequently-updated cases don't fall too far behind.

Let me know if this should be cross-posted in the forum for more discussion!

[John Lenton (chipaca)]

I thought we'd explicitly said the license needed to be SPDX 2.something, and that bumping the spdx version was a when-and-if discussion we needed to have at some point.

I also thought we'd agreed to use the exact same validation code on the store and in snapd.

[Daniel Manrique (roadmr)]

Looks like *this* is the "bumping the spdx version" discussion, and the point we're having it at is the point when users are starting to use non-SPDX 2.x identifiers and complaining about it.

Clearly stating "we only support SPDX 2.x identifiers" is a possible solution. I'm not opposed to it and if you think that's reasonable that can be done, which would, I think, result on the snapd bug (this) being closed and the store bug being updated to say it's about clarifying which version we support.

Using the same validation code would not have helped the user who reported this; that's something we can discuss separately. The forum thread https://forum.snapcraft.io/t/keeping-up-with-spdx-license-identifier-updates/7095/4 mentions that as a possibility but in order for it to help keep up with SPDX updates, other changes are also needed.

Let me know if you think mentioning "only accept SPDX 2.x identifiers and be clear about it" as a solution in the forum thread makes sense, it's a bit repeaty but some people follow snapd bugs and some follow the forum, so it doesn't seem out of the question.

Thanks for your reply!

[Michael Vogt (mvo)]

This is being worked on in https://github.com/snapcore/snapd/pull/6152