Provide a flag for a controller to retry invalid cloud credentials

[Thomas Cuthbert]

Our Juju (2.8.7) controller was not picking up changed public IP addresses for controller machines from the cloud provider (gce).

On investigation it was discovered that the controller's credentials were marked invalid.

Comparing the credentials stored on the controller with the credentials that were used to bootstrap the environment showed no differences.

Given the credentials were always valid, one can conclude that the cloud provider's API was returning false negatives, causing Juju to mark the credentials invalid.

It would be nice if Juju had configurable flags for us to auto-recover from this type of event. If we could do something like:

  retry-credentials: true
  retry-credentials-interval: 1h

It would reduce operational toil having to manually update credentials for each of our environments.

[Haw Loeung (hloeung)]

This is still happening, even with controllers running 2.9.42 (as seen today with the jaas-aws-us-east-1-001 controller)