Juju provisioner server error API mismatch after upgrading MAAS from 1.9 to 2.4.2

[Drew Freiberger]

We have a site which has undergone an upgrade of the MAAS service from 1.9 series to 2.x series on xenial.

Once upgraded, we found that we had to reauthenticate to the maas CLI api with a new endpoint with a v2.0 URL due to this error:

ServerError: 410 GONE (The 1.0 API is no longer available. Please use API version 2.0.)

The URL went from http://192.168.0.2/MAAS/api/1.0/ to /MAAS/api/2.0/ and worked after that.

This is also coming out of the juju controller logs on machine-0 at a pace since we tried deploying two new machines.

When I attempt to juju update-cloud maas -f clouds.yaml --controller maas, I get the error:

juju update-cloud maas -f ./clouds.yaml --controller maas
ERROR updating controller cloud is not supported by this version of Juju.

My controller is at version 2.4.1, and my juju binary is 2.5.2 (snapped).

Is this update-cloud available in a later version of juju?
Is there a method we can use to update the api URL for the maas provider in the controller model?

[Richard Harding (rharding)]

This seems another case of https://bugs.launchpad.net/juju/+bug/1819456 where update-cloud is intended to fix but there's some final work going into making sure everything restarts and works appropriately.

[Ian Booth (wallyworld)]

The ability to update a cloud in a controller is not available in Juju 2.4.
The 2.5.3 release will allow that for Openstack.
MAAS will be supported in the upcoming 2.6 release in May, and possibly the next 2.5.x release if we do one.
The only way to do it now is to edit the database directly - the "clouds" collection - and restart the controller agents.

[David O Neill (dmzoneill)]

hey

@wallyworld

Drew may already have this information.

But could you provide further information on this edit, in the example of schema, queries updates.

Thanks

[Ian Booth (wallyworld)]

The information for any clouds managed by the controller is stored in the "clouds" collection.
Once connected to the Juju mongo database, there's some simple commands you can run:

db.clouds.find().pretty()

will show that's there, including field names so you can see what can be updated.

The _id value for a cloud is used when updating it. So to change the endpoint for a particular region, you can do something like:

db.clouds.update({_id:"mycloud"}, {$set: {"regions.us-east-1.endpoint": "https://someurl"}})

The "mycloud" and "us-east-1" above are replaced by your actual cloud values. It may be that there's a global endpoint for the cloud instead of region specific ones. So in that case you'd do:

db.clouds.update({_id:"mycloud"}, {$set: {"endpoint": "https://someurl"}})

The output of the find() command should make it clear what other attributes there are. Use dot notation to drill down to nested fields as shown in the region example above.

[Ian Booth (wallyworld)]

We'll land a fix for 2.5.4
https://github.com/juju/juju/pull/9960

[Hans van den Bogert (hbogert)]

Wow this hit me hard.. everybody coming from older MAAS version is probably still on port 80 through Apache.
This should've been a big red banner in the release notes.

Actually, what was the correct way for someone to update to 2.4 icw Juju? Was this an oversight for the 2.4 release? MAAS and Juju go hand in hand, nobody went through this with testing?

[Tim Penhey (thumper)]

This was probably an oversight from the perspective of Juju documentation.

We had made the assumption that most of the deployments coming from MAAS 1.9 would end up being redeployed, and those that weren't were being handled by sustaining engineering.

Dealing with an upgraded MAAS requires two things:
 1. Updating the underlying MAAS endpoint to talk to
 2. Updating the instance IDs used by Juju to identify the machines

In the cases where we were dealing with this and Juju 2.4 we were updating the database directly with Javascript though the mongo shell. Not awesome, but it got the job done.

There are internal documents somewhere about the process. However since it was not something that was generally considered as used publicly, there wasn't the effort to document it publicly. Sorry.