juju 2.2.2.1 unable to add a unit to an existing service, no matching agent binaries found

[David Lawson]

We're in the process of trying to add arm64 units to one of our Scalingstacks, we have one unit that's been in the cloud since its initial deployment. I've recently adding several more units to the MaaS but attempting to deploy them gives me a down machine with the error "no matching agent binaries available". I'm not seeing an enormous amount of information in the controller logs even with debugging enabled, this seems to be largely the extent of it:

2017-09-01 14:18:56 DEBUG juju.environs.tools tools.go:101 finding agent binaries in stream "devel"
2017-09-01 14:18:56 DEBUG juju.environs.tools tools.go:105 reading agent binaries with major version -1
2017-09-01 14:18:56 DEBUG juju.environs.tools tools.go:111 filtering agent binaries by version: 2.2.2.1
2017-09-01 14:18:56 DEBUG juju.environs.tools tools.go:114 filtering agent binaries by series: xenial
2017-09-01 14:18:56 DEBUG juju.environs.tools tools.go:117 filtering agent binaries by architecture: arm64
2017-09-01 14:18:56 DEBUG juju.environs.tools urls.go:109 trying datasource "keystone catalog"
2017-09-01 14:18:56 DEBUG juju.environs.simplestreams simplestreams.go:683 using default candidate for content id "com.ubuntu.juju:devel:tools" are {20161007 mirrors:1.0 content-download streams/v1/cpc-mirrors.sjson []}

That looks like it's trying to do the right thing, I never see an error from this process until the final no matching agent binaries found. The model config is all defaults. I believe this was deployed with 2.2.2 and upgraded to 2.2.2.1, it's possible it's looking for a 2.2.2.1 agent binary and not finding it, I notice there's no matching version for that in the streams.

[John A Meinel (jameinel)]

I'm not sure what the best resolution is. The issue is that someone upgraded to a 'manually build' binary/local jujud. When doing so, that would only upload the version on your machine (thus amd64 only). And there are no binaries matching for arm64. Hence the recommendation to not use '--upload-tools' style workflow.

I believe there would be ways to build binaries for other platforms and get them uploaded, we'd have to dig into the best way to do that. A different way would be to allow you to "juju upgrade-juju" from 2.2.2.1 back to a 2.2.2 which would have official binaries.

[David Lawson (deej)]

I discussed this further with Nick Skaggs and dug through the command histories a little more, the upgrade was performed without --upload-tools so if a binary was uploaded, juju did it on its own. The environment was bootstrapped without --upload-tools and shows that the original binaries came from streams. It does appear that the upgraded binaries were uploaded, but that decision appears to have happened inside the juju client for reasons now lost to the mists of time. I would venture to guess that there's another bug similar to https://bugs.launchpad.net/juju/+bug/1696563 lurking in the client code somewhere.

[David Lawson (deej)]

I've verified that this is not a recurrence of 1696563, both the client machines and controller can access streams.c.c via http and https both, so this is another failure mode.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 5 years, so we're marking it Expired. If you believe this is incorrect, please update the status.