Juju set-credential for a new credential on the controller does not work, you need to restart the agent on the controller.

[Pedro Victor Lourenço Fragola]

[Description]

I have Juju 2.6.10 with a local Cloud with MAAS, I have two Juju credentials for MAAS each one as an API KEY when I set the new Juju credential and remove the old MAAS key Juju keeps using the old one being necessary to restart the agent on the controller.

[Reproduction]

Juju - 2.6/stable

Controller MAAS
maas-cred1 - key=> 5FtZTPdqFg9FWvPMfQ:Z6WdH9YbLuqDHSuYha:XUjYW7q7hrF3vwJMU65acuyaVxmjVV6H
maas-cred2 - key=> 29An2FDPbVcsswnRQT:rn6LqpTzT5LZVNBPmc:HUapZgNWqmFZJYUY7mE3pWd2vd9eTyC7

Cloud Credentials
maas maas-cred1, maas-cred2

juju show-model
default:
  name: admin/default
  short-name: default
  model-uuid: c135a081-8c0a-4314-8113-a948d0de1507
  model-type: iaas
  controller-uuid: f87a0b32-0a52-4cdd-8fe4-ea940b3c8683
  controller-name: maas
  is-controller: false
  owner: admin
  cloud: maas
  type: maas
  life: alive
  status:
    current: available
    since: "2020-05-26"
  users:
    admin:
      display-name: admin
      access: admin
      last-connection: 41 seconds ago
  machines:
    "22":
      cores: 1
  sla: unsupported
  agent-version: 2.6.10
  credential:
    name: maas-cred1
    owner: admin
    cloud: maas

Set new credential with new API KEY

juju set-credential maas maas-cred2
Did not find credential remotely. Looking locally...
Uploading local credential to the controller.
Changed cloud credential on model "admin/default" to "maas-cred2".
administrator@homer:~$ juju update-credential maas maas-cred2
Credential valid for:
  default
Controller credential "maas-cred2" for user "admin" on cloud "maas" updated.
For more information, see ‘juju show-credential maas maas-cred2’.

At this time I removed the maas-cred1 credential from MAAS and Juju

juju remove-credential maas-cred1

[Impact]

juju add-machine
created machine 23
juju status
Model Controller Cloud/Region Version SLA Timestamp
default maas maas 2.6.10 unsupported 10:48:20-03:00

Machine State DNS Inst id Series AZ Message
22 started 172.16.99.6 sure-midge bionic default Deployed
23 down pending bionic could not get environ: Authorization Error: 'Invalid access token: Z6WdH9YbLuqDHSuYha'

At this moment juju still has reference to old credential and I need to restart the agent on the controller.

juju ssh ubuntu@172.16.99.3 "sudo systemctl restart jujud-machine-0.service"
Connection to 172.16.99.3 closed.

juju add-machine
created machine 24
Machine State DNS Inst id Series AZ Message
22 started 172.16.99.6 sure-midge bionic default Deployed
23 down pending bionic could not get environ: Authorization Error: 'Invalid access token: Z6WdH9YbLuqDHSuYha'
24 pending 172.16.99.7 united-fox bionic default Deploying: Powering on

Now this works without a problem, I believe this is not the normal behavior of Juju. I already had this behavior with connection to vSphere.

[Pen Gale (pengale)]

Running this w/ 2.8, things behaved a bit differently, in that I got a warning that the old credentials were still attached to the "default" model.

```
petevg@badjanet:~$ juju remove-credential guimaas maas-test-01
This operation can be applied to both a copy on this client and to the one on a controller.
Do you want to remove credential "maas-test-01" for cloud "guimaas" from:
    1. client only (--client)
    2. controller "guimaas-default" only (--controller guimaas-default)
    3. both (--client --controller guimaas-default)
Enter your choice, or type Q|q to quit: 3
Found remote cloud "guimaas" from the controller.
Found local cloud "guimaas" on this client. Credential "maas-test-01" for cloud "guimaas" removed from this client.
ERROR could not remove remote credential: cannot revoke credential cloudcred-guimaas_admin_maas-test-01: it is still used by 1 model
```

I'm not certain that's expected behavior. Will follow up with the team, then follow up on this bug.

[Ian Booth (wallyworld)]

Credential handling in 2.6 had various issues that have been addressed in 2.8

Juju will not let you remove a credential if doing so would break a model due to that credential being used (that's the expected behaviour in comment #1). This check wasn't done in 2.6 from memory.

With needing to restart the agent in 2.6, that's possible also as work was done in 2.8 to make updating things such as credentials and cloud info visible to the agents without a restart.

I suggest upgrading to 2.8 as short of critical security fixes, there won't be any more 2.6 releases.

[Pen Gale (pengale)]

@wallyworld: is there a procedure for updating the credentials for a model? If so, is that documented somewhere? (If not, we should probably document ...)

[Ian Booth (wallyworld)]

The doc for credentials is here. I think it's mostly up to date but suspect it may need updating to reflect the latest tweaks to the commands over the last cycle

https://discourse.juju.is/t/tutorial-managing-credentials/1289

[Pedro Victor Lourenço Fragola (pedrovlf)]

I upgraded to version 2.8 and after some tests, I have this scenario:

[Reproduction]
I changed my controller and models to use the maas-cred2 credential:

$juju set-credential maas maas-cred2
Found credential remotely, on the controller. Not looking locally...
Changed cloud credential on model "admin/default" to "maas-cred2".
$ juju set-credential maas maas-cred2 -m controller
Found credential remotely, on the controller. Not looking locally...
Changed cloud credential on model "controller" to "maas-cred2".

Then I removed the maas-cred1 credential that was being used:

$ juju remove-credential maas maas-cred1
This operation can be applied to both a copy on this client and to the one on a controller.
Do you want to remove credential "maas-cred1" for cloud "maas" from:
    1. client only (--client)
    2. controller "maas" only (--controller maas)
    3. both (--client --controller maas)
Enter your choice, or type Q|q to quit: 3
Found remote cloud "maas" from the controller.
Found local cloud "maas" on this client.
Credential "maas-cred1" for cloud "maas" removed from this client.
Credential "maas-cred1" for cloud "maas" removed from the controller "maas".

[Impact]
I tried to add a new machine and got the error:
juju add-machine
created machine 42
$ juju status
Model Controller Cloud/Region Version SLA Timestamp
default maas maas 2.8.0 unsupported 15:39:29-03:00

Machine State DNS Inst id Series AZ Message
41 started 172.16.99.7 proper-yeti bionic default Deployed
42 down pending bionic retrieving environ: cloud credential "maas/admin/maas-cred1" not found (not found)

[workaround]
$ juju ssh -m controller 0 "sudo systemctl restart jujud-machine-0.service"
Connection to 172.16.99.3 closed.

$ juju add-machine
created machine 44
juju status
Model Controller Cloud/Region Version SLA Timestamp
default maas maas 2.8.0 unsupported 20:56:27-03:00

Machine State DNS Inst id Series AZ Message
41 started 172.16.99.7 proper-yeti bionic default Deployed
42 down pending bionic retrieving environ: cloud credential "maas/admin/maas-cred1" not found (not found)
44 pending 172.16.99.6 epic-locust bionic default Deploying: Loading ephemeral

Really in version 2.8 when I have not yet set the new credential it is not possible to delete the current credential ... however, when I set the new credential and delete the old it still has some reference and it only works after a restart the agent, I do not know this happens in an installation from scratch with 2.8, because I did an upgrade.

[Ian Booth (wallyworld)]

This is weird - the services used to provision resources in the cloud watch for credential changes and update their local cloud clients to use any new credential when updated. It seems something is going wrong with the MAAS provider in trying to achieve this.

[Canonical Juju QA Bot (juju-qa-bot)]

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.