The other day I was rolling out some monitoring services to the JAAS controllers and I noticed that "juju expose" seemed to have no effect. This turned out to be because the controller had suspended the controller model itself: "suspended since cloud credential is not valid".
I ran "juju update-credential $cloud $credential" and the message went away and the newly exposed ports became accessible. I checked again today and two of the controller models are suspended again. Today's victims are an Azure controller running 2.6.5, and an AWS controller running 2.6.6. It's also happened with GCE controllers running 2.6.6.
I've checked the logs and the only message that mentions credentials is a bunch of:
WARNING juju.api.credentialvalidator backend.go:84 cloud credential reference is set for the model but the credential content is no longer on the controller
One of the controllers also reports "credential cloudcred-azure_[REDACTED]@external_hl-az will be deleted but it is used by model [REDACTED]" but this is clearly not the controller's credential and so therefore I assume is unrelated.
I've taken a look at the cloudCredentials collection. Azure isn't too helpful:
juju:PRIMARY> db.cloudCredentials.find({owner: "admin"}).pretty()
{
"_id" : "azure#admin#jaas-prodstack-cdo-azure",
"owner" : "admin",
"cloud" : "azure",
"name" : "jaas-prodstack-cdo-azure",
"revoked" : false,
"auth-type" : "service-principal-secret",
"attributes" : {
"application-id" : "[REDACTED#1]",
"application-password" : "[REDACTED#2]",
"subscription-id" : "[REDACTED#3]"
},
"txn-revno" : NumberLong(14),
"txn-queue" : [ ],
"invalid" : true,
"invalid-reason" : "azure cloud denied access"
}
juju:PRIMARY> _
AWS is more verbose, but the credentials absolutely work, so I'm puzzled why it's in this state:
juju:PRIMARY> db.cloudCredentials.find({owner: "admin"}).pretty()
{
"_id" : "aws#admin#jaas",
"owner" : "admin",
"cloud" : "aws",
"name" : "jaas",
"revoked" : false,
"auth-type" : "access-key",
"attributes" : {
"access-key" : "[REDACTED#1]",
"secret-key" : "[REDACTED#2]"
},
"txn-revno" : NumberLong(28),
"txn-queue" : [ ],
"invalid" : true,
"invalid-reason" : "\nThe provided credentials could not be validated and \nmay not be authorized to carry out the request.\nEnsure that your account is authorized to use the Amazon EC2 service and \nthat you are using the correct access keys. \nThese keys are obtained via the \"Security Credentials\"\npage in the AWS console.\n: AWS was not able to validate the provided access credentials (AuthFailure)"
}
juju:PRIMARY> _
I dug through the AWS controller's logs and found some AuthFailure errors, although they were for instances that are not in the controller model, so I assume they're unrelated to this problem.
All credenitals that are stored on the controller are called 'remote' or 'controller' credentials. Controllers do not actually use credentials, only models do. Controllers only store credentials.
If the models share a credential and that credential is deemed by a cloud provider as 'invalid', i.e. a model starts getting auth errors from cloud calls, then Juju will mark credential as invalid and ALL models that are using it will be suspended.
There are several scenarios that need to be considered here. If your models are sharing a credential that is valid for some models but is not valid for other, then you should be using different credentials in these models. You can 'set-model-
If the credential becomes invalid and you have run 'update-
I am happy to talk you through individual instances. Feel free to reach out on IRC.
I think that Juju is behaving as expected here.