autopkgtests after shutdown of atlas depend on vagrant 2.x

[Christian Ehrhardt ]

autopkgtest issue in vagrant-mutate

These tests are fairly new in vagrant-mutate (1.2.0-3)
  * Add DEP-8 smoke test
by Antonio Terceiro <email address hidden> on Thu, 02 Feb 2017 11:47:08

These tests currently block:
- qemu
- distro-info

Examples bad run in [1][2]

It is interesting that it seemed to work on the trigger by:
- ruby-default
- vagrant

Example good run in [3][4]

This only does a conversion, so it works on all arches.
It does not kick the actual virt which would be too arch dependent.

mutate is a vargant plugin and lists correctly in all cases.
Checking global-status shows a clean env in all cases.

Then the box add fails in some cases:
Good case:
+ vagrant box add --force --provider virtualbox debian/stretch64
==> box: Loading metadata for box 'debian/stretch64'
    box: URL: https://vagrantcloud.com/debian/stretch64
==> box: Adding box 'debian/stretch64' (v9.3.0) for provider: virtualbox
    box: Downloading: https://vagrantcloud.com/debian/boxes/stretch64/versions/9.3.0/providers/virtualbox.box

Bad case:
+ vagrant box add --force --provider virtualbox debian/stretch64
The box 'debian/stretch64' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:
URL: ["https://atlas.hashicorp.com/debian/stretch64"]
Error: The requested URL returned error: 404 Not Found

+ vagrant box add --force --provider virtualbox debian/stretch64
The box 'debian/stretch64' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:
URL: ["https://atlas.hashicorp.com/debian/stretch64"]
Error: The requested URL returned error: 404 Not Found

This pretty much seems like the recent atlas -> cloud change that Odd bloke was fixing.
See bug 1747426
TL;DR of this is:
- DEFAULT_SERVER_URL = "https://atlas.hashicorp.com"
+ DEFAULT_SERVER_URL = "https://vagrantcloud.com"

The issue itself is reproducible in a bionic container.
This seems to match now - this is fixed in vagrant 2.0.2+dfsg-2
So this test worked, but the others installed 1.9.1+dfsg-1 and failed.
[4]: is special as it is the oldest test - there atlas.hashicorp.com wasn't down yet.

vagrant-mutate itself would not need a dependency to newer vagrant, but actually the tests would (as they were pushed post the change).

In Debian there is no vagrant 1.9.x anymore - so it is not an issue for them.

We could make complex uploads (to later resync) to add a dependency to the test.
But actually I think we should mark it as badtest (on this version only) which would resolve it for now.

OTOH that might kill later test coverage if this is the versions that will stay in bionic.
IIRC there is a way to tell the CI to test this against the newer vagrant.
So like a retry with special info.
I need to find how that works.

[1]: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/v/vagrant-mutate/20180207_112502_c9d0e@/log.gz
[2]: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/v/vagrant-mutate/20180205_122808_bbdea@/log.gz

[3]: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/v/vagrant-mutate/20180203_182842_557fb@/log.gz
[4]: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic/bionic/amd64/v/vagrant-mutate/20180201_234554_405c1@/log.gz

[Christian Ehrhardt  (paelzer)]

I discussed with Pitti, via adding the extra trigger of vagrant this will work.
So we will retry the tests originally triggered by distro-info and qemu but with &trigger=vagrant%2F2.0.2%2Bdfsg-2 appended.

That would force it to pin the tested + vagrant from proposed and that should work.
Full list of links:

Qemu
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=amd64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu1&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=arm64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu1&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=armhf&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu1&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=i386&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu1&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=ppc64el&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu1&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=s390x&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu1&trigger=vagrant%2F2.0.2%2Bdfsg-2
Distro-info
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=amd64&package=vagrant-mutate&trigger=distro-info%2F0.18&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=arm64&package=vagrant-mutate&trigger=distro-info%2F0.18&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=armhf&package=vagrant-mutate&trigger=distro-info%2F0.18&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=i386&package=vagrant-mutate&trigger=distro-info%2F0.18&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=ppc64el&package=vagrant-mutate&trigger=distro-info%2F0.18&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=s390x&package=vagrant-mutate&trigger=distro-info%2F0.18&trigger=vagrant%2F2.0.2%2Bdfsg-2

Note: It is important that there are no others hitting the "normal" retry button as this will be queued and mark it bad again.

[Christian Ehrhardt  (paelzer)]

Vagrant 2 still did not migrate.
So it blocks qemu again - new links to trigger the test of vagrant-mutate (done for qemu) against the new working vagrant version.

https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=amd64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu2&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=arm64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu2&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=armhf&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu2&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=i386&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu2&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=ppc64el&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu2&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=s390x&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu2&trigger=vagrant%2F2.0.2%2Bdfsg-2

[Christian Ehrhardt  (paelzer)]

Again ...

https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=amd64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=arm64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=armhf&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=i386&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=ppc64el&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=s390x&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2

[Christian Ehrhardt  (paelzer)]

Due to ruby transition we need all proposed to get ruby in all of its variety for 2.5.
Hopefully the rest in proposed is not having issues, utherwise this will become a very long list of triggers.

https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=amd64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=arm64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=armhf&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=i386&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=ppc64el&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=s390x&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&trigger=vagrant%2F2.0.2%2Bdfsg-2&all-proposed=1

[Christian Ehrhardt  (paelzer)]

Since we are using all proposed for ruby I don't need to explicitly select vargant.
Also the recent rebuild made the old trigger target unavailable.

https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=amd64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=arm64&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=armhf&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=i386&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=ppc64el&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&all-proposed=1
https://autopkgtest.ubuntu.com/request.cgi?release=bionic&arch=s390x&package=vagrant-mutate&trigger=qemu%2F1%3A2.11%2Bdfsg-1ubuntu4&all-proposed=1