Support other distributions as sources (Fedora, Mageia, openSUSE, Debian) instead of Ubuntu

[Neal Gompa]

The snapcraft tool (and the yaml format) currently mandate Ubuntu repositories as the primary binary input source (that is, the base of snaps created via snapcraft is always Ubuntu).

However, if I want to create a snap using Fedora, CentOS, Mageia, openSUSE, or even Debian as the binary package base, I cannot currently.

There are two particular problems with this:
1) The modern python3-apt API is unavailable for RPM based distributions because apt shipped in RPM based distros is so old that it doesn't work on it. While apt in RPM based distributions does support Debian packages/repositories, it also supports RPM ones. However, since the version of apt is very old (0.5), it doesn't work with the modern python-apt bindings package. This forces snapcraft to only run on Debian based systems.

2) Snapcraft on Debian based systems will always use Ubuntu as the base, which may be undesirable for people who'd like to use Debian software to build their snaps.

My suggestion is to support RPM based distributions by using gir bindings for libdnf[1] or the Python 3 API for DNF[2]. The major RPM based Linux distributions (Fedora, Mageia, CentOS, openSUSE) are able to use libhif/DNF. Fedora uses it by default now, and Mageia will offer it as a secondary package manager with Mageia 6[3]. It's available as a COPR for CentOS[4], and it's not much trouble to make it available for openSUSE, too.

Add a stanza where the user can set the distro base, and snapcraft will use the appropriate engine and repo configuration to construct the snap.

Obviously, some place where the distro repo information can be stored as snapcraft config is important too. With the existing code, being able to swap Ubuntu and Debian distro base would be nice.

[1]: https://github.com/rpm-software-management/libdnf
[2]: http://dnf.readthedocs.io/en/latest/api.html
[3]: https://wiki.mageia.org/en/Mageia_6_Release_Notes#Package_management
[4]: https://copr.fedorainfracloud.org/coprs/g/rpm-software-management/dnf-stack-el7/

Tasks for completing this:

1. Add support for individual RPMs as source - LP: #1640296 - GH#snapcore/snapcraft#870 - DONE

2. Refactor build/stage packages to support multiple backends:
   - LP: #1673448 GH#snapcore/snapcraft#1192 - DONE
   - LP: #1673449 GH#snapcore/snapcraft#XXXX - IN PROGRESS

3. Add support for different distro base snaps and refactor core snap to depend on base snap - LP#??? - GH#snapcore/snapd#??? - NOT STARTED

4. Add support for declaring desired base in snapcraft - LP#??? - GH#snapcore/snapcraft#??? - NOT STARTED

5. Add RPM+DNF backend for supporting Fedora, CentOS, Mageia, and OpenSUSE bases - LP#1602258 - GH#snapcore/snapcraft#??? - NOT STARTED

6. Run integration tests in all the supported operating systems.

[Mark Shuttleworth (sabdfl)]

Seems sensible to enable folks to use whatever distro they prefer, thank
for the guidance.

 status: confirmed

[Neal Gompa (ngompa13)]

From the discussion at the Snap sprint:

* Snapcraft will be reworked to support using different backends for build-packages and stage-packages. While currently it has a hard dependency on python3-apt, the plan is to add a libhif backend for RPM based distributions once the rework is complete. The libhif backend will work for the major RPM based distributions (Fedora, openSUSE, Mageia, etc.) because they use a common repository format. Since libhif itself has minimal dependencies (librpm, librepo, and libsolv), is used in Fedora and Mageia, a build has been made in openSUSE through the OBS[1], and offers Python bindings through gobject-introspection, it should work for our purposes.

* Snapcraft.yaml will grow an "on <distro>" stanza that will indicate what distribution base is to be used. This way, Snapcraft preserves the reproducibility of snaps by ensuring that the snap production is tied to the distribution base. This will enable using Ubuntu or Debian as a snap distro base using the apt backend, while using libhif for Fedora, Mageia, openSUSE, etc.

[1]: https://build.opensuse.org/package/show/home:M0ses:snappy/libhif

[Tim Jester-Pfadt (t-jp)]

"* Snapcraft.yaml will grow an "on <distro>" stanza that will indicate what distribution base is to be used."

Does that mean that snapcraft.yaml becomes distro specific? It would be best to avoid that so that you can run "snapcraft snap" on any snapcraft.yaml on any distro and get the same output. Unfortunately, I see only two ways to achieve that. 1) Add per distro build-packages, stage-packages lists with the right package names used by the distro into the snapcraft.yaml. Problem here is that the yaml file grows very long and nothing guarantees that it covers all distros. 2) Build packages using containers by default. This way you could build a snapcraft.yaml that takes a Fedora base and build it on Ubuntu, by setting up a temporary Fedora container first and the other way around.

[Neal Gompa (ngompa13)]

No, it means that regardless of distribution you are running, snaps will be able to be constructed with the source information that the author of the snapcraft.yaml wants to use, from any distribution supported by snapcraft.

[Neal Gompa (ngompa13)]

I've been meaning to post here on a status update on this, but I keep forgetting...

Anyway, at the Snap sprint in October, we discussed this again, this time delving a bit more into the details.

The plan is still the same as comment #2 indicates.

I have taken the first steps towards RPM support by adding support for using individual RPMs as sources[1], mimicking the support for using individual DEBs as sources. This will roll out with Snapcraft 2.22.

The next step will be to refactor the {build,stage}-packages to have apt/Ubuntu code split out into its own module, set up in a way that it implements a generic distro backend class. The class-oriented structure will make it easy to add other backends. Once this is in place, we will start implementing a backend using python3-dnf (part of dnf[2]), as it's more high level and stable than libdnf at the moment.

As for the snapcraft.yaml, we were leaning towards using the "for" keyword to indicate the distribution base, but I'm starting to think that maybe "use" or "using" might be better keywords, as it clearly indicates the expected relationship. We aren't completely certain yet of how we will indicate the release version (for example, Fedora 25, Mageia 6, openSUSE 42.2, etc.), as apparently "series" is special and we might need a different way to represent this. The current thought is to express it as a list (e.g. use: ["fedora", 25]) or a key-value map (e.g. use: [dist: "mageia", rel: 6]). Again, though, we're not quite at the point where we are able to start quibbling about the exact syntax.

However, we *are* making progress!

[1]: https://bugs.launchpad.net/snapcraft/+bug/1640296
[2]: https://github.com/rpm-software-management/dnf

[Sergio Schvezov (sergiusens)]

Thanks for summarizing Neil, I guess the next step is to make snapcraft.internal.repo pluggable.

[Sergio Schvezov (sergiusens)]

err Neal!

[Sergio Schvezov (sergiusens)]

https://github.com/snapcore/snapcraft/pull/1192

[Neal Gompa (ngompa13)]

@Sergio,

This bug should not get closed by the completion of that PR, though it is one of the biggest hurdles.

[Sergio Schvezov (sergiusens)]

On Wed, 15 Mar 2017 02:16:57 +0000, Neal Gompa wrote:
> @Sergio,
>
> This bug should not get closed by the completion of that PR, though it
> is one of the biggest hurdles.

It would be good to get a list of things left to be done for it to be completed :-)

--
Sent using Dekko from my Ubuntu device

[Neal Gompa (ngompa13)]

The list of items to actually complete this ticket (with status next to them)

1. Add support for individual RPMs as source - LP#1640296 - GH#snapcore/snapcraft#870 - DONE

2. Refactor build/stage packages to support multiple backends - LP#??? - GH#snapcore/snapcraft#1192 - IN PROGRESS

3. Add support for different distro base snaps and refactor core snap to depend on base snap - LP#??? - GH#snapcore/snapd#??? - NOT STARTED

4. Add support for declaring desired base in snapcraft - LP#??? - GH#snapcore/snapcraft#??? - NOT STARTED

5. Add RPM+DNF backend for supporting Fedora, CentOS, Mageia, and OpenSUSE bases - LP#1602258 - GH#snapcore/snapcraft#??? - NOT STARTED

[Sergio Schvezov (sergiusens)]

thanks, I am moving this to the description so it doesn't get lost

[Zygmunt Krynicki (zyga)]

I'm marking this as in-progress in snapd since there is on-going work to support multiple base snaps used by various app snaps at the same time.

[Neal Gompa (ngompa13)]

There is in progress work to get DNF (and thus python3-dnf) into openSUSE Factory: https://build.opensuse.org/request/show/517054

Once it is in, we'll have a single API we can use for snapcraft to use for all RPM-based distributions.

[Neal Gompa (ngompa13)]

DNF is now in openSUSE, so we have a common API for the RPM backend for all supported RPM-based Linux distributions.

[Sergio Schvezov (sergiusens)]

Abstraction of source management has been implemented in snapcraft a while back.

What remains on snapcraft's side is for someone to add a :
- DNF backend.
- a base and build environment for that base.

[Zygmunt Krynicki (zyga)]

I'm marking the snapd side of this bug as invalid. Snapd supports using arbitrary base snaps already but is not in the business of making snap packages. I think the snapcraft task models this problem correctly and should remain open.