Individual instance of merged agent lost password

[Wouter van Bommel]

Running a juju model version 2.9.27 we have the following situation:

Model Controller Cloud/Region Version SLA Timestamp
stg-snap-store prodstack-is-beta prodstack5/prodstack5 2.9.27 unsupported 06:03:36Z

App Version Status Scale Charm Store Rev OS Notes
filebeat 6.8.1 active 0/1 filebeat local 1 ubuntu
nrpe active 0 nrpe jujucharms 75 ubuntu
snapident-r1d3f1da 1d3f1da active 1 snapident local 13 ubuntu
telegraf active 0 telegraf jujucharms 44 ubuntu exposed

Unit Workload Agent Machine Public address Ports Message
snapident-r1d3f1da/0* active idle 650 10.132.2.184 8001/tcp ready
  filebeat/448 unknown lost 10.132.2.184 agent lost, see 'juju show-status-log filebeat/448'
  nrpe/652 active idle 10.132.2.184 icmp,5666/tcp Ready (source version/commit cs-nrpe-...)
  telegraf/604* active idle 10.132.2.184 9103/tcp Monitoring snapident-r1d3f1da/0 (source version/commit 26e531a)

The server is running one of the versions in which juju combined all the server side agents. So on this server a `sudo systemctl restart jujud-machine-650.service` was executed.
This resulted in the same status as mentioned above.

The file /var/log/juju/unit-filebeat-448.log contains the following:
2022-04-14 13:38:14 INFO juju.worker.logger logger.go:136 logger worker stopped
2022-04-14 13:38:14 INFO juju.worker.uniter uniter.go:317 unit "filebeat/448" shutting down: catacomb 0xc0004ca000 is dying
2022-04-14 13:38:35 INFO juju unit_agent.go:277 Starting unit workers for "filebeat/448"
2022-04-14 13:39:03 ERROR juju.worker.apicaller connect.go:209 Failed to connect to controller: invalid entity name or password (unauthorized access)

So the password of this individual unit has expired on a machine that is only running a single agent for all the units?

Output of `juju show-status-log filebeat/448`
Time Type Status Message
13 Apr 2022 04:20:22Z juju-unit executing running beats-host-relation-joined hook for snapident-r1d3f1da/0
13 Apr 2022 04:20:28Z juju-unit executing running beats-host-relation-changed hook for snapident-r1d3f1da/0
13 Apr 2022 04:20:29Z juju-unit idle
14 Apr 2022 11:35:41Z juju-unit executing running leader-settings-changed hook
14 Apr 2022 11:36:04Z juju-unit idle
14 Apr 2022 12:13:06Z juju-unit executing running leader-settings-changed hook
14 Apr 2022 12:13:12Z juju-unit idle
14 Apr 2022 12:19:05Z juju-unit executing running leader-settings-changed hook
14 Apr 2022 12:19:38Z juju-unit idle
14 Apr 2022 12:49:18Z juju-unit executing running leader-settings-changed hook
14 Apr 2022 12:49:22Z juju-unit idle
14 Apr 2022 12:50:59Z juju-unit executing running leader-settings-changed hook
14 Apr 2022 12:51:16Z juju-unit idle
14 Apr 2022 12:54:59Z juju-unit executing running leader-settings-changed hook
14 Apr 2022 12:55:10Z juju-unit idle
14 Apr 2022 13:08:41Z juju-unit executing running leader-settings-changed hook
14 Apr 2022 13:09:08Z juju-unit idle
14 Apr 2022 13:25:07Z juju-unit executing running leader-settings-changed hook
14 Apr 2022 13:25:09Z juju-unit idle
14 Apr 2022 13:35:12Z workload active Filebeat ready.

[Heather Lanigan (hmlanigan)]

Please see:

https://bugs.launchpad.net/juju/+bug/1956975/comments/5 for a log message request.

https://bugs.launchpad.net/juju/+bug/1956975/comments/6 for a way to resolve.

Note, we've had some cases where the workaround doesn't work. However we do not have enough data to understand why at this point.