upgrading juju controllers from snap fails to find tools

[Dmitrii Shcherbakov]

ubuntu@maas:~$ snap list
Name Version Rev Developer Notes
core 16-2 1577 canonical -
juju 2.1.2 1042 canonical classic
lxd 2.12 1743 canonical -

ubuntu@maas:~$ juju upgrade-juju -m controller
no prepackaged tools available, using local agent binary 2.1.2.1
ERROR no matching tools available

Succeeds when a version is specified manually:

ubuntu@maas:~$ juju upgrade-juju -m controller --agent-version 2.1.2
started upgrade to 2.1.2

Fun times.

[Anastasia (anastasia-macmood)]

Please provide logs :)

[Dmitrii Shcherbakov (dmitriis)]

Should be somewhere around UTC 13:53:09 though there are time jumps in the log:

2017-04-11 13:53:09 DEBUG juju.worker.dependency engine.go:478 "upgrade-steps-runner" manifold worker started
2017-04-11 13:53:09 DEBUG juju.wrench wrench.go:112 couldn't read wrench directory: stat /var/lib/juju/wrench: no such file or directory
2017-04-11 13:53:09 DEBUG juju.worker.dependency engine.go:492 "upgrade-steps-runner" manifold worker stopped: <nil>

The previous version was not a snapped one if it makes any difference.

On success the message was as follows:

➜ /tmp grep 'starting upgrade from' machine-0.log
2017-04-11 14:02:30 INFO juju.worker.upgradesteps worker.go:349 starting upgrade from 2.0.2 to 2.1.2 for "machine-0"

[Curtis Hovey (sinzui)]

What is the current version of the controller show in "juju status -m controller"?

What do the agent-stream and agent-metadata-url show in "juju model-config -m controller"?

I suspect the controller thinks it is a devel or local version (2.0.2.1) instead of a relased version. You appear to want your client and server locked to the same version. I always upgrade using --agent-version because Juju will abort if it cannot do exactly that. Otherwise, juju might choose a version you didn't expect.

[Tim Penhey (thumper)]

Pretty sure this is the lookup logic. It isn't overly great with the defaults.

[Dmitrii Shcherbakov (dmitriis)]

Sorry about the lack of feedback, the issue was intermittent and I did not have a good reproducer at the time.

[Dmitrii Shcherbakov (dmitriis)]

Got an update from 2.1.2 to 2.1.3:

(openstack-client) dmitriis@maas-master:~/ha-bundle⟫ snap info juju | grep installed
installed: 2.1.3 (1922) 24MB classic

juju status -m controller
Model Controller Cloud/Region Version Notes
controller mightymaas mightymaas 2.1.2 upgrade available: 2.1.3

App Version Status Scale Charm Store Rev OS Notes

Unit Workload Agent Machine Public address Ports Message

Machine State DNS Inst id Series AZ
0 started 10.0.7.0 kawm3e xenial default

juju model-config -m controller | grep agent
agent-metadata-url default ""
agent-stream default released
agent-version model 2.1.2

Same issue on upgrade:

juju upgrade-juju -m controller
no prepackaged tools available, using local agent binary 2.1.3.1
ERROR no matching tools available

[Tim Penhey (thumper)]

My best guess right now is a failure on the streams lookup. Transient network error or something.

There is another bug out there that says something along the lines of "if the controller says there is an upgrade available of a particular version, and the user says "upgrade-juju", it should fail if it can't get said version".

[Ian Booth (wallyworld)]

If this is running on openstack, the issue is that agent metadata would have been added to keystone, pointing to agents stored in a swift container. This location is in the search path before streams.canonical.com. Once Juju finds agent metadata in any search path segment, it uses that metadata and looks no further. This is by design to prevent unwanted agent binaries from accidentally being used. The real issue here is that the keystome metadata and agents are out of date.

[Dmitrii Shcherbakov (dmitriis)]

{Ian, Tim},

It was a MAAS controller.

Regarding the network issue: adding --agent-version 2.1.3 solved this right away while upgrade-juju without this argument kept failing.

[Paul Gear (paulgear)]

If the symptoms @dmitrii mentioned above are part of this bug, then I don't think lp:1694374 is a duplicate, because the symptoms in that case were that --agent-version=2.1.3 had no effect.

[Dmitrii Shcherbakov (dmitriis)]

Well, now I cannot do anything at all.

ubuntu@maas:~/juju/cmd$ snap info juju
name: juju
summary: "juju client"
publisher: canonical
contact: http://jujucharms.com
description: |
  Through the use of charms, juju provides you with shareable, re-usable, and
  repeatable expressions of devops best practices.
commands:
  - juju
tracking: beta
installed: 2.2-rc1 (1925) 25MB classic
refreshed: 2017-06-05 21:52:07 -0400 EDT
channels:
  latest/stable: 2.1.3 (1922) 24MB classic
  latest/candidate: 2.1.3 (1922) 24MB classic
  latest/beta: 2.2-rc1 (1925) 25MB classic
  latest/edge: 2.2-rc1+develop-7256fe0 (1915) 44MB classic

ubuntu@maas:~/juju/cmd$ juju upgrade-juju -m controller
no prepackaged tools available, using local agent binary 2.2-rc1.1
ERROR no matching agent binaries available

ubuntu@maas:~/juju/cmd$ juju upgrade-juju -m controller --agent-version 2.2-rc1
ERROR no matching agent binaries available

[John A Meinel (jameinel)]

So I think, 2.2-rc1 ends up producing agents that are in the 'proposed'
(devel?) stream, rather than the 'released' stream. I also believe that
official snaps of the juju client do not produce 'jujud' binaries inside
the snap, because there is an official jujud*.tar.gz that can be downloaded
from streams.canonical.com (--edge based releases do carry the jujud
binary, hence the 44MB vs 25MB).

However, controllers that are currently on 'stable' releases don't see
'proposed' streams as something they can target as an upgrade. It might be
possible to do something with "juju model-config -m controller
agent-stream=?", but its something I haven't personally tested.

Can you confirm whether you can/cannot bootstrap a new controller with the
2.1-rc1 snap?

John
=:->

On Tue, Jun 6, 2017 at 4:29 PM, Dmitrii Shcherbakov <
<email address hidden>> wrote:

> Well, now I cannot do anything at all.
>
> ubuntu@maas:~/juju/cmd$ snap info juju
> name: juju
> summary: "juju client"
> publisher: canonical
> contact: http://jujucharms.com
> description: |
> Through the use of charms, juju provides you with shareable, re-usable,
> and
> repeatable expressions of devops best practices.
> commands:
> - juju
> tracking: beta
> installed: 2.2-rc1 (1925) 25MB classic
> refreshed: 2017-06-05 21:52:07 -0400 EDT
> channels:
> latest/stable: 2.1.3 (1922) 24MB classic
> latest/candidate: 2.1.3 (1922) 24MB classic
> latest/beta: 2.2-rc1 (1925) 25MB classic
> latest/edge: 2.2-rc1+develop-7256fe0 (1915) 44MB classic
>
>
> ubuntu@maas:~/juju/cmd$ juju upgrade-juju -m controller
> no prepackaged tools available, using local agent binary 2.2-rc1.1
> ERROR no matching agent binaries available
>
> ubuntu@maas:~/juju/cmd$ juju upgrade-juju -m controller --agent-version
> 2.2-rc1
> ERROR no matching agent binaries available
>
> --
> You received this bug notification because you are subscribed to juju.
> Matching subscriptions: juju bugs
> https://bugs.launchpad.net/bugs/1681853
>
> Title:
> Juju Tools Upgrade from 2.0.2 to 2.1.2 fails: ERROR no matching tools
> available
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/juju/+bug/1681853/+subscriptions
>

[Dmitrii Shcherbakov (dmitriis)]

I've destroyed the controller with 2.2beta4 and successfully bootstrapped a new one with 2.2-rc1 (with the juju snap):

ubuntu@maas: $ juju status -m controller
Model Controller Cloud/Region Version SLA
controller samaas samaas 2.2-rc1 unsupported

App Version Status Scale Charm Store Rev OS Notes

Unit Workload Agent Machine Public address Ports Message

Machine State DNS Inst id Series AZ Message
0 started 10.232.4.171 8tk7tc xenial default Deployed

[Samuel Cozannet (samuel-cozannet)]

I had the same sort of issues with 2.2.1:

$ juju upgrade-juju
no prepackaged tools available, using local agent binary 2.2.1.1
ERROR no matching agent binaries available

Fixed with providing --agent-version=2.2.1 but was weird.

[Dmitrii Shcherbakov (dmitriis)]

Guys, it's perfectly reproducible:

$ git clone https://github.com/juju/juju.git && cd juju
$ git checkout juju-2.1.2
HEAD is now at 7f85784... Merge pull request #7094 from hmlanigan/bug-1671265

$ snapcraft
$ sudo snap install juju_2.1.2_amd64.snap --classic --dangerous

$ juju bootstrap <cloud_name>

$ sudo snap remove juju
$ sudo snap install juju # 2.2.1 is the latest at the time of writing

$ juju upgrade-juju -m controller
no prepackaged tools available, using local agent binary 2.2.1.1

$ juju upgrade-juju -m controller --agent-version 2.2.1
started upgrade to 2.2.1

[Xav Paice (xavpaice)]

Just saw a similar thing with 2.2.3 to 2.2.4. I was getting a slightly different message, but adding --agent-version 2.2.4 allowed the upgrade to work. I think in this case it was looking for 2.2.4.1?

https://pastebin.canonical.com/198601/

Error I saw (for search engine) was "ERROR no matching agent binaries available".

[Chris Sanders (chris.sanders)]

I've just seen this with 2.2.5.1 as well. Installed via snap, running 2.2.4 receive the error
"no prepackaged tools available, using local agent binary 2.2.5.1
ERROR no matching agent binaries available"

providing the --agent-version of 2.2.5 appears to be working

[Sandor Zeestraten (szeestraten)]

Hitting the same here trying to upgrade from 2.2.4 to 2.2.5

# Snap info
zeestrat@vagrant:~/dev/charms/charm-slurm-node$ sudo snap info juju
name: juju
summary: juju client
publisher: canonical
contact: http://jujucharms.com
description: |
  Through the use of charms, juju provides you with shareable, re-usable, and
  repeatable expressions of devops best practices.
snap-id: e2CPHpB1fUxcKtCyJTsm5t3hN9axJ0yj
commands:
  - juju
tracking: stable
installed: 2.2.5 (2713) 25MB classic
refreshed: 2017-10-20 06:07:42 +0000 UTC
channels:
  stable: 2.2.5 (2713) 25MB classic
  candidate: 2.2.6+2.2-d19dc5b (2717) 43MB classic
  beta: 2.3-beta1 (2640) 29MB classic
  edge: 2.3-beta2+develop-f5763bb (2722) 52MB classic

# Juju version
zeestrat@vagrant:~$ juju version
2.2.5-xenial-amd64

# Model version
zeestrat@vagrant:~$ juju status
Model Controller Cloud/Region Version Notes SLA
controller lxd lxd 2.2.4 upgrade available: 2.2.5 unsupported

App Version Status Scale Charm Store Rev OS Notes

Unit Workload Agent Machine Public address Ports Message

Machine State DNS Inst id Series AZ Message
0 started 10.225.81.44 juju-c79fb8-0 xenial Running

# Trying to upgrade controller model
zeestrat@vagrant:~$ juju upgrade-juju
no prepackaged tools available, using local agent binary 2.2.5.1
ERROR no matching agent binaries available