autopkgtests for 2.23.1+17.04 have regressed in release

[Steve Langasek]

The autopkgtests for snapd 2.23.1+17.04 have regressed since that package released zesty, presumably because of changes in external services (store contents?) that the snapd autopkgtests depend on.

http://autopkgtest.ubuntu.com/packages/s/snapd/zesty/amd64

I have marked the autopkgtests to be ignored for the current version. This test failure will become a blocker for the next update of snapd.

It would be ideal if someone could pin down the source of the regression (e.g. update of the core snap?) and ensure that the same autopkgtests are used to gate updates on that side.

[Steve Langasek (vorlon)]

I spoke with pedronis about this. It appears the snapd package in the devel series tests against the edge versions of the snaps, and the core snap is currently broken on edge.

https://github.com/snapcore/snapd/blob/2.23.1/packaging/ubuntu-16.04/tests/integrationtests#L27

The above code picks stable or candidate channel for snaps based on the pocket of the snapd package. This is definitely wrong for the devel series, where the package is in neither -proposed nor -updates, but rather in the release pocket.

I think it's also wrong for SRUs. Whether the autopkgtest is triggered for a new snapd or a new package that snapd depends on (random example: snapd), this is a CI test for the updated *package*, and the version of the core snap used should not be varied.

If the autopkgtests are also being used in a different context to validate updates of the core snap itself (which sounds like a good idea to me), this use of the edge channel should be configured in some other way independent of the snapd package's pocket.