Provide way for admins of controllers to remove models from other users (in particular suspended)

[Haw Loeung]

Hi,

We operate public JAAS (Juju-as-a-Service) controllers for community consumption. Sadly, a lot of these models are in "suspended" or "destroying" state and the logs shows we no longer have valid credentials to access them. E.g. for one of the Azure JAAS controllers:

"""
2023-03-08 04:41:09 ERROR juju.worker.dependency engine.go:693 "environ-tracker" manifold worker returned unexpected error: creating environ for model "start" (a42357b6-7ec3-4d48-8bd6-b90fde70e0b1): getting tenant ID: expected unauthorized error response, got 404: GET https://management.azure.com/subscriptions/3faffd88-5277-4b86-b9fa-1cbde6b60f38
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: SubscriptionNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "SubscriptionNotFound",
    "message": "The subscription '3faffd88-5277-4b86-b9fa-1cbde6b60f38' could not be found."
  }
}
"""

"""
2023-03-08 04:41:16 ERROR juju.worker.dependency engine.go:693 "environ-tracker" manifold worker returned unexpected error: creating environ for model "test1" (d292e17a-...): ClientSecretCredential authentication failed
POST https://login.microsoftonline.com/e5eba1d1-.../oauth2/v2.0/token
--------------------------------------------------------------------------------
RESPONSE 401 Unauthorized
--------------------------------------------------------------------------------
{
  "error": "invalid_client",
  "error_description": "AADSTS7000222: The provided client secret keys for app '60a04dc9-...' are expired. Visit the Azure portal to create new keys for your app: https://aka.ms/NewClientSecret, or consider using certificate credentials for added security: https://aka.ms/certCreds.\r\nTrace ID: 2a954576-...\r\nCorrelation ID: ff18f90e-...\r\nTimestamp: 2023-03-08 04:41:16Z",
  "error_codes": [
    7000222
  ],
  "timestamp": "2023-03-08 04:41:16Z",
  "trace_id": "2a954576-...",
  "correlation_id": "ff18f90e-...",
  "error_uri": "https://login.microsoftonline.com/error?code=7000222"
}
--------------------------------------------------------------------------------
"""

Can we please have a command to forcefully remove these? We don't care about the state of VMs or resources provisioned by these models, we just care that they're removed from the Juju controller and no longer being managed. With all these requests to Azure, I'm not surprised when they'll eventually rate limit or block these JAAS controllers access to talk to their API.

[Ian Booth (wallyworld)]

As a workaround, as a controller "superuser" role, are you able to grant yourself "admin" on the model? Then you could do it.

[Haw Loeung (hloeung)]

Sadly, I tried with granting myself as the "admin" on the model without any luck:

| juju grant admin admin <redacted>@external/test
| juju destroy-model <redacted>@external/test --force --no-wait
| ...
| Waiting for model to be removed.................................................
| ...
| Because the destroy model operation did not finish, there may be cloud resources left behind.
| ERROR timeout after 30m0s

With the model still around.

[Ian Booth (wallyworld)]

AH, you had the ability to actually invoke the destroy-model without a "not allowed" error. That's what I thought the bug was referring to. You are saying that you to to run the destroy but juju does not want to actually remove the model due to issues with the cloud etc. --force should result in the model being gone IMO.

[Harry Pidcock (hpidcock)]

https://github.com/juju/juju/pull/15328