new primary wrongfully updates replication password on leadership change

[Romain Couturat]

postgres unit leadership changed after a juju controller upgrade.
The new primary updated "_juju_repl" user password (aka replication_password) in the primary DB but not in it's .pgpass file, nor did it update anything on the secondary and broke the replication (the secondary having deprecated creds)
It was fixed by applying replication_password from .pgpass (identical on both unit) again on the primary.

afaiu :
{
from juju logs : https://pastebin.canonical.com/p/PHz5fPTV9v/ (not public)
"Invoking reactive handler: reactive/postgresql/client.py:50:set_client_passwords"
updates the replication password and breaks the replica.
It shouldn't (there is already a password) but _juju_repl user pw is not matched.

Either because it's not an entry in "client_passwords" (but an entry on it's own https://pastebin.ubuntu.com/p/srt7D9sTry/)
and/or because replication username is hardcoded here
https://git.launchpad.net/postgresql-charm/tree/reactive/postgresql/postgresql.py#n156
to a different value than replication_username from
https://git.launchpad.net/postgresql-charm/tree/reactive/postgresql/replication.py#n340
}

[Romain Couturat (romaincout)]

by "but _juju_repl user pw is not matched." I mean
https://git.launchpad.net/postgresql-charm/tree/reactive/postgresql/client.py#n52
appears to set the password again failing to see it already exists

[Romain Couturat (romaincout)]

happened to me again today, might also be caused by adding a unit with a newer version version of the charm than the one that was previously used to deploy the first unit(s)