snapcraft prerequites are slow to resolve

[Michael Hall]

In large projects with many parts, resolving prerequisites can be extremely slow. I have attached the snapcraft.yaml for building kdevelop. After a full build and without cleaning, I ran "snapcraft stage" under a profiler. This was essentially a noop command, but the prerequisite checking took more than 10 minutes.

Also attacked is the output from "snapcraft stage" which show only prerequisite resolution and skipping steps. Also attached is the profiler output showing the top 100 lines in terms of time used, as well as the top 100 function calls by count.

[Scarlett Gately Moore (scarlettmoore)]

I don't know how much it will help, but I am taking a different approach, and adding a misc part and adding all the build-packages for all the parts there so that apt will only be used for the misc, qt, and frameworks parts, rather than for every part. This will all be better when snaps can use libraries from other snaps as then I will build a snap for each framework and we only need to pull in the required pre-built frameworks snaps that are needed. When I tried to put each framework as a part it was a spectacular level of fail as the dependency chain was enormous.

[Mark Shuttleworth (sabdfl)]

I think we can plan to make a snap of the KDE framework stack which is
maintained by Ubuntu and can safely be used by other snaps. We don't yet
have the content-sharing interface, but as soon as we do, we can make a
kde-base snap which provides a full set of libraries that apps can just use.

Generally we avoid cross-vendor sharing of libraries, but in the case of
something fundamental like KDE it makes sense to trust the community
process to produce something of high quality that everyone can depend
on. Apps can individually choose how much of that they link to
dynamically, vs static linking or library bundling in their own snaps.

Mark

On 09/06/16 09:54, Scarlett Gately Clark wrote:
> I don't know how much it will help, but I am taking a different
> approach, and adding a misc part and adding all the build-packages for
> all the parts there so that apt will only be used for the misc, qt, and
> frameworks parts, rather than for every part. This will all be better
> when snaps can use libraries from other snaps as then I will build a
> snap for each framework and we only need to pull in the required pre-
> built frameworks snaps that are needed. When I tried to put each
> framework as a part it was a spectacular level of fail as the dependency
> chain was enormous.
>

[Kyle Fazzari (kyrofa)]

Nice, I love bug reports that really push the limits of the system. Snapcraft should obviously be handling this better; barring the library snap discussion, snaps are supposed to include their dependencies. Snapcraft should not be unwieldy when trying to do that for a large system. I'll investigate this.

Thanks for the great bug Michael-- I appreciate the profile as well.

[Sergio Schvezov (sergiusens)]

https://github.com/snapcore/snapcraft/pull/794