stage-packages does not handle unmet dependencies

[ryeterrell]

We're attempting to build snaps for multiple platforms from a single box (so a single platform, amd64). We're doing this with pre-built binaries
and the dump plugin.

We've gotten things to mostly work using the `architectures` yaml tag, but we hit a snag when we need to use `stage-packages` within a part. Instead of pulling down the platform-specific package and dependencies, it appears to pull them down for the current arch. In the following gist, the snap was built against amd64 followed by ppc64le: http://bit.ly/2oMRbLb

There you can see that it's ignoring amd64 debs, but of course it should instead (I think) be attempting to download ppc64le debs.

[Kyle Fazzari (kyrofa)]

You can use syntax like `stage-packages: ['hello:<arch>']`, but the builder will need to be configured to pull debs from that arch (`dpkg --add-architecture`).

[George Kraft (cynerva)]

@kyrofa Thanks for the proposed workaround. I'm not having any luck building an arm64 snap from amd64, though.

I did `dpkg --add-architecture amd64` as suggested, and also updated /etc/apt/sources.list to add ports.ubuntu.com:

deb [arch=amd64,i386] http://archive.ubuntu.com/ubuntu xenial main restricted
deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports xenial main restricted
...

After this, I'm able to download arm64 packages with e.g. `apt download ceph-common:arm64`.

However, when trying to build a snap with `stage-packages: ['ceph-common:arm64']`, snapcraft silently omits the ceph-common:arm64 package and its dependencies.

[George Kraft (cynerva)]

Oops, typo there. I did `dpkg --add-architecture arm64`, not amd64.

[Kyle Fazzari (kyrofa)]

Huh, indeed, confirmed. `apt_cache['ceph-common:arm64']` appears to succeed, but then it's not included in `apt_cache.get_changes()` so it's never actually fetched. I'm not sure what apt is doing, there, we'll need to investigate.

[Kyle Fazzari (kyrofa)]

Oops. `apt_cache['ceph-common:arm64'].mark_install()` appears to succeed, but then it's not included in `apt_cache.get_changes()`

[Nick Zatkovich (nzatkovich)]

Has there been any movement on this since it was reported?

[Kyle Fazzari (kyrofa)]

Not directly, but we've been revamping that section of code, particularly relating to architectures. We need to revisit after the refactor.

[ryeterrell (ryeterrell)]

This is still broken in the same way as of snapcraft v2.34.

[Cris Dywan (kalikiana)]

On master the following works for me:

parts:
    test:
        stage-packages:
            - htop:armhf
        snap:
            - usr/bin/htop

The armhf version of htop ends up in the snap.

[George Kraft (cynerva)]

Still no good for ceph-common:s390x. I can confirm that htop:s390x works for me using snapcraft from master, but doing the same with ceph-common:s390x, none of the contents are included in the snap.

[Kyle Fazzari (kyrofa)]

@Christian: No one is debating that using <package>:<arch> generally works. But it's been pointed out (and confirmed) that a specific package isn't working the way, with no errors. Do you have any insight into why this is happening with ceph-common?

[Nick Zatkovich (nzatkovich)]

@Christian: any movement on this?

[Cris Dywan (kalikiana)]

It appears that apt.Cache()[foobar].mark_install() doesn't raise an exception in case of a problem with a package, instead silently changing marked_keep to True and leaving marked_install set to False, which results in get_changes() not returning any packages - so as a result Snapcraft silently proceeds without the package and without an error message.
In this particular case python-requests:armhf can't be installed. For comparison, apt-get will do this:
$ sudo apt-get install ceph-common:armhf
Reading package lists... Done
Building dependency tree
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 ceph-common:armhf : Depends: librbd1:armhf (= 10.2.9-0ubuntu0.16.04.1) but it is not going to be installed
                     Depends: python-cephfs:armhf (= 10.2.9-0ubuntu0.16.04.1) but it is not going to be installed
                     Depends: python-rados:armhf (= 10.2.9-0ubuntu0.16.04.1) but it is not going to be installed
                     Depends: python-rbd:armhf (= 10.2.9-0ubuntu0.16.04.1) but it is not going to be installed
                     Depends: python-requests:armhf but it is not installable
                     Depends: librados2:armhf (>= 0.87) but it is not going to be installed
                     Depends: libradosstriper1:armhf (>= 0.93) but it is not going to be installed
                     Depends: librgw2:armhf (>= 10.2.9) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

I proposed a PR introducing a new error class and code to detect this case: https://github.com/snapcore/snapcraft/pull/1788

[Kyle Fazzari (kyrofa)]

Nice to have this be an error instead of silently failing, but what is the actual problem with the package/dependencies? We're just downloading these and unpacking them, we don't need apt to be happy about it.

[Cris Dywan (kalikiana)]

Let me try and elaborate on that: if you try and install like above apt-get behaves as if python-requests didn't exist on armhf. You can install it without a suffix just fine. This may become clearer if you check https://packages.ubuntu.com/xenial/python-requests or "apt-cache show python-requests" which reveals that it's "Architectures: all", except it doesn't declare multi-arch behavior [1] which means the package is effectively not available at all.

So at this point we're talking about working around packaging issues, which may be worth our while, but we'll need to be careful since we're deviating from what apt-get does.

[1] https://wiki.ubuntu.com/MultiarchSpec#Dependencies_involving_Architecture:_all_packages

[Cris Dywan (kalikiana)]

Some notes from brainstorming this with Kyle just now:
1. Ideally the packaging should be fixed.
2. Theoretically we should be able to detect the case of a package being "all" without the multi-arch declaration and getting it in a non-multi-arch way, thus enabling more packages without involving maintainers and SRU's.
2. We may long term be able to have a cache per architecture so that resolving no longer depends on multi-arch. I'll need to check the Python API again and see if that idea is feasible.

[Cris Dywan (kalikiana)]

I reported bug 1737973 to address the first point.

[Kyle Fazzari (kyrofa)]

Hey folks, knowing what the issue is here, when you run into it, try using `stage-packages: [ceph-common:all]`. It seems to work.

[George Kraft (cynerva)]

Thanks for investigating this.

I'm not having much luck with `stage-packages: [ceph-common:all]`. This just gives me amd64 binaries for e.g. <snap-root>/usr/bin/rbd - am I missing something that goes with this?