attempting to delete locked maas node results in juju marking valid cloud credentials as invalid

[Steven Parker]

Replicate with MAAS node that has been locked.

Remove node with:
    juju remove-machine and/or --force

juju status will yield
  foundations-maas maas_cloud 2.9.18 unsupported 22:01:30Z suspended since cloud credential is not valid

Workaround:
  juju update-credentials maas_cloud maas_cloud_credentials
  and/or
  juju set-credential -m lma maas_cloud maas_cloud_credentials

NOTE: Loss of cloud credentials can lead to actions such as add-machine, remove-machine failing to complete or error out. These actions simply hang. In fact remove-machine --force will also fail.

[Ian Booth (wallyworld)]

If Juju makes an cloud (MAAS in this case) API call, if the cloud responses with codes 401, 403 etc then Juju will deem the credential to be invalid as the api call was rejected as unauthorised.

It seems that here MAAS might be responding to attempt to delete a locked node with 403, thus triggering the juju to deem the credential as invalid. We need to audit what return codes MAAS uses and see what's appropriate to use to trigger the invalid credential behaviour in juju.

[Steven Parker (sbparke)]

Further work around if needed is

If nodes are out of the model you'll get an error

Credential invalid for:
  openstack:
    couldn't find instance "bga76y" for machine 14
    couldn't find instance "h3hbrd" for machine 13
Failed models may require a different credential.
Use ‘juju set-credential’ to change credential for these models before repeating this update.

So you'll need to force in this case.

juju update-credentials maas_cloud maas_cloud_credentials --force

[John Lettman (jplettman)]

It would be a nice improvement for Juju to warn that the machine is locked.