Established units continue to change their password

[Nicolas Vinuesa]

When deploying, once the new unit is started, the following logs can be seen:

$ juju debug-log -m m --replay --tail | grep password
machine-0: 12:51:14 DEBUG juju.worker.apicaller connecting with old password
machine-0: 12:51:14 DEBUG juju.worker.apicaller changing password...
machine-0: 12:51:14 INFO juju.worker.apicaller [1c93c5] password changed for "machine-0"
machine-0: 12:51:14 DEBUG juju.worker.apicaller connecting with current password
machine-0: 12:51:14 TRACE juju.rpc.jsoncodec -> {"request-id":3,"type":"Agent","version":3,"request":"SetPasswords","params":{"changes":[{"tag":"unit-prometheus2-0","password":"***/***"}]}}
unit-prometheus2-0: 12:51:14 INFO juju.worker.apicaller [1c93c5] password changed for "unit-prometheus2-0"
machine-0: 12:51:14 TRACE juju.rpc.jsoncodec -> {"request-id":3,"type":"Agent","version":3,"request":"SetPasswords","params":{"changes":[{"tag":"unit-prometheus2-0","password":"***/***"}]}}

There should not be need for this so this change is an anomally.

[Harry Pidcock (hpidcock)]

This is how juju ensures the password it was initially given by some other means remains private (e.g. the password was in the agent config file that was put on the instance via instance metadata).

[John A Meinel (jameinel)]

So this isn't actually about "New Units" because that is meant to happen. (A new unit *should* see that it had to log in with 'old-password' and then negotiate a new password over the TLS channel).

The issue is that an existing, stable unit continues to keep changing its password each time it logs in again.
I'll tweak the wording a bit to be clearer about what we want to fix.

[John A Meinel (jameinel)]

That said, in trying to create a clean reproducer, it seems that the issue is just one about logging, not one about actually changing the password. These are the steps I tried:

juju bootstrap lxd lxd --debug
juju add-model test
juju deploy jameinel-ubuntu-lite
juju debug-log -m controller # in one window

juju ssh ubuntu-lite/0
sudo -i
cd /var/log/juju/agents/unit-ubuntu-lite-0
root@juju-6e4447-0:/var/lib/juju/agents/unit-ubuntu-lite-0# grep password agent.conf
apipassword: Auub3/pAtxaQKSRuBrC/nv5T
oldpassword: AvdsckVEF8hj06Yd6ZYlSK/t

root@juju-6e4447-0:/var/lib/juju/agents/unit-ubuntu-lite-0# juju_stop_unit ubuntu-lite/0
ubuntu-lite/0: stopped
root@juju-6e4447-0:/var/lib/juju/agents/unit-ubuntu-lite-0# juju_start_unit ubuntu-lite/0
ubuntu-lite/0: started

In the other window I can see the log lines for:
machine-0: 11:19:09 INFO juju.apiserver.connection agent login: unit-ubuntu-lite-0 for 714a0816-ce6b-4be4-8c7c-e838126e4447
machine-0: 11:19:09 INFO juju.apiserver.common setting password for "unit-ubuntu-lite-0"

However:
root@juju-6e4447-0:/var/lib/juju/agents/unit-ubuntu-lite-0# grep password agent.conf
apipassword: Auub3/pAtxaQKSRuBrC/nv5T
oldpassword: AvdsckVEF8hj06Yd6ZYlSK/t

It didn't actually change the passwords. We should also confirm that inside of Mongo it also has the same password hash for the unit. (I did this multiple times and can see that it continually has the same password as before, but every juju_stop_unit+juju_start_unit triggers the same 'setting password' log entry)

But the fix for this is to fix the log messages to only log when we *actually* change the password for the unit.