Recovery tarballs for snap builds

Bug #1763639 reported by Christian Dywan on 2018-04-13
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
High
Colin Watson
launchpad-buildd
High
Colin Watson

Bug Description

Launchpad should have a mechanism to fetch all sources required to build a snap and store the result of it. A branch can then be created from it to allow future re-builds of the same snap and applying patches on top of the branch as necessary.

This would be achieved by essentially separating out the "snapcraft pull" step during the build, which retrieves all external dependencies, and creating a copy of the result for later use.

Related branches

Colin Watson (cjwatson) wrote :

We need to specify this in a great deal more detail; it's not actionable until we do. Things I can think of right now include:

 * Is the "branch" part truly necessary, as opposed to simply creating a tarball that somebody can then later unpack and drop into a branch if necessary? The latter is a lot easier - we don't have many of the mechanisms we'd need to create a branch at this point, and although they could no doubt be built, there are many other non-obvious things here such as how to specify the branch ownership, whether it should be Bazaar or Git, etc.
 * snapcraft.yaml often specifies remote branches / artifacts / etc. to fetch. How would we suppress this behaviour for a subsequent rebuild? Would we need to produce an edited version of snapcraft.yaml?
 * The output of "snapcraft pull" isn't likely to be anyone's preferred form for modification. Have there been any experiments done on the snapcraft side to work out what the workflow would actually look like for developers? I think this sort of thing needs to come before implementation in launchpad-buildd.
 * Exactly what files and directories need to be saved from the "snapcraft pull" step in order that snapcraft won't try to re-fetch them in a later step? What must we, or indeed developers working with the output, avoid doing so that snapcraft won't decide that it needs to pull again?

Changed in launchpad-buildd:
status: New → Incomplete
Christian Dywan (kalikiana) wrote :

I misused the term branch here. On the Launchpad side, only a tarball should be created and stored. I'll rename the bug accordingly. Actually creating a VCS branch from that is optionally done by a person when and if that is necessary.

This mechanism is indeed meant for recovery purposes only. Not something to be maintained frequently but if there's no better option than to apply an urgent patch on the existing sources. Pulling in a new version of the same snap might contain any number of changes. By the same token, patching and re-building the same snap from source may pull in different dependencies or rely on external sources no longer available.

The result of "snapcraft pull" is actually to be taken as-is here. The entire folder should be copied. After running it, all artifacts will be contained in the project tree. When building from this copy, Snapcraft can detect that all dependencies are already satisfied and will build from it without reaching out to the network to retrieve them. No modifications to the snapcraft.yaml are therefore necessary.

Changed in launchpad-buildd:
status: Incomplete → New
summary: - Recovery branches for snap builds
+ Recovery tarballs for snap builds
description: updated
Colin Watson (cjwatson) wrote :

OK. Is there any preference for what this tarball should be called? Should the tarball always be created, or is there some reasonable condition we can use to decide whether to create it?

Christian Dywan (kalikiana) wrote :

The condition would be that it's a stable snap (you wouldn't apply security patches to unstable releases). However it should probably be enabled explicitly to limit the amount of data for snaps that don't make use of it.

Colin Watson (cjwatson) on 2018-04-20
tags: added: feature lp-snappy
Changed in launchpad:
status: New → Triaged
importance: Undecided → High
Changed in launchpad-buildd:
status: New → Triaged
importance: Undecided → High
Colin Watson (cjwatson) on 2018-04-20
Changed in launchpad-buildd:
status: Triaged → In Progress
assignee: nobody → Colin Watson (cjwatson)
Colin Watson (cjwatson) on 2018-04-20
Changed in launchpad:
status: Triaged → In Progress
assignee: nobody → Colin Watson (cjwatson)
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
Colin Watson (cjwatson) wrote :

2018-04-22 07:34:41,648 INFO 2209-83-2 applied just now in 0.2 seconds

tags: added: qa-ok
removed: qa-needstesting
Launchpad QA Bot (lpqabot) wrote :
tags: added: qa-needstesting
removed: qa-ok
Changed in launchpad:
status: In Progress → Fix Committed
Colin Watson (cjwatson) on 2018-04-23
tags: added: qa-ok
removed: qa-needstesting
Changed in launchpad-buildd:
status: In Progress → Confirmed
Colin Watson (cjwatson) on 2018-04-23
Changed in launchpad-buildd:
status: Confirmed → In Progress
Colin Watson (cjwatson) wrote :

launchpad-buildd (161) xenial; urgency=medium

  * Pass build URL to snapcraft using SNAPCRAFT_IMAGE_INFO.
  * Add an option to generate source tarballs for snaps after pulling
    external dependencies (LP: #1763639).

 -- Colin Watson <email address hidden> Mon, 23 Apr 2018 09:44:49 +0100

Changed in launchpad-buildd:
status: In Progress → Fix Released
Colin Watson (cjwatson) on 2018-04-24
Changed in launchpad:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers