snapd bundles golang dependencies despite being in main

[Michael Terry]

It is security and MIR team policy to ensure that statically built golang packages in main use archive packages for their golang dependencies rather than bundling them in.

When putting snappy into main for xenial (see bug 1548887), there was a lot of work to properly depend on archive packages. That work seems to have regressed and it now appears that the snapd package is bundling all dependencies in as vendor modules. It looks like this bundling started in 2.16 (Sep 2016).

You can find some documentation on the policies:
 - http://bazaar.launchpad.net/~ubuntu-security/ubuntu-cve-tracker/master/view/head:/README.built_using
 - https://wiki.ubuntu.com/MIRTeam
 - http://pkg-go.alioth.debian.org/packaging.html

I'm guessing that the bundling was motivated by trusty support, which may not have the requisite archive packages in it? But we should have a discussion about the resulting policy breakage and what to do about it. I don't believe that discussion has happened yet.

Besides the support issues, this also means that snapd trunk can't be built directly (e.g. in a PPA).

[Zygmunt Krynicki (zyga)]

Snapd is a complex application that is constantly evolving (new features are being added in nearly every release made) and it is unreasonable to create something like snapd without depending on a new library or new version of a library as the need arises. Bringing each and every dependency back to 14.04, 16.04, 16.10 and 17.04 on an on-going basis is unrealistic as it would easily consume the time of a full-time engineer that we cannot allocate at this time. During the way the Debian and Ubuntu policy is structured this would also delay (possibly each new release) by weeks at a time.

Snapd has been granted exceptional SRU process in recognition of the unique role it plays in stable distributions and the unique shift in process it represents. While one could easily do everything "the old way", packaging each small golang library, going through weeks in NEW or in SRU (for every Ubuntu release around) this would severely compromise our mission to deliver a fantastic new package manager to everyone. At the same time you must realize that it would just be a specific class of busy work. There's no final difference if snapd copies a .go file into the vendor tree (which is managed automatically with go vendor), into some other directory or if we put it in Ubuntu. This all results in the exact same effect. We choose to recognize this fact and spend our resources resourcefully to maximize the effect our efforts have on the world.

One more significant argument is on the QA effort. A snap (not snapd) delivered to 14.04, 16.04, 16.10 and any other (non-ubuntu distribution) is the exact same binary object. QA effort spent on one release has non-zero, valuable measure on every other release. To maintain our aggressive release schedule we at the very least wish to build the exact same source everywhere (we cannot yet use the exact same binary but that would be even better for the reason outlined above).

I hope that this explanation makes sense and that we can all see the big picture. Having said that I would like to work with you to improve whatever we can as long as it doesn't severely stretch our resources.

[Tyler Hicks (tyhicks)]

My thoughts on this are that it may be acceptable to bundle dependencies in SRUs when the stable Ubuntu release doesn't already have the given dependency packaged. This would be acceptable to me in snapd's case since upstream has a very close relationship with Ubuntu Security, we trust that they'll be helpful in performing any necessary security updates, and they regularly SRU new upstream releases.

However, I'd like to see dependencies not be bundled in the current development release of Ubuntu. I think there will overlap between dependencies of projects such as snapd, lxd, juju, etc., and I'd prefer that those dependencies be packaged up and re-used to ease the maintenance burden.

This also makes it possible for the Ubuntu Security team to more accurately track security issues in dependencies of snapd. Our current CVE triage process is more effective when archive packages exist for projects affected by CVEs. Note that this will likely change in the future as we see an increased need to identify and assist in the tracking of security issues in Canonical supported snaps.

[Michael Hudson-Doyle (mwhudson)]

I think Tyler's proposal makes sense. I maintain snapd with unbundled dependencies in Debian sid, and it's really not been very painful.

One thing we want is for snapd to fairly closely track new versions of its dependencies; any security fixes are likely to be developed for the tip of the dependency fixed, and we want it to be easy to backport any fixes to the version snapd is using. The proposal is a step in that direction. (Ironically, a test failed in snapd in debian builds because one dependency is newer in Debian than the version snapd vendors).