HA Juju controllers showing inconsistent status

[James Simpson]

Hi,

We've experienced an issue with the consistency and stability of our Juju controllers, and are struggling to pinpoint what's actually happening.

We're operating a HA controller set, running Juju 2.9.42, deployed in an Openstack cloud.

Symptoms we've observed have been:

* Issues with the stability of relationship hooks in deployed models (we have observed issues with relationships being created, updated, and departed)

* Controllers returning inconsistent "juju status" results
  When running "juju status --debug" to make sure we get one result from each controller, we have observed that at least one controller will consistently return a different result than the other(s).
  For example, this paste shows both secondary controllers reporting the primary controller as "agent-lost", while the primary disagrees: https://pastebin.ubuntu.com/p/VHrwFbm79Z/

Controller logs from the period in question have been made available via secure portal https://juju-controller-reports.admin.canonical.com/ps5-prodstack-is/

Model logs for the specific model in which we observed relationship hook issues are located in "special-request" under that directory.

Please advise if there are any additional logs we should supply, any metrics we can gather from the time, or anything else.

Thanks!

[Ian Booth (wallyworld)]

The controllers share agent connectivity info (aka presence) using pubsub. I don't think there's an explicit delivery guarantee for such messages.

Logging to turn on would be

juju.worker.pubsub=TRACE
juju.worker.presence=TRACE

Logs could also contain messages matching the format string

"%p programming error, e.ch=%v did not accept %v - missing Unwatch?\nwatch source:\n%s"

Extra relation debug can be obtained by setting

juju.worker.uniter.relation=TRACE

[James Simpson (jsimpso)]

Thanks Ian - I've configured those via "juju model-config -m controller logging-config=...", next time we experience an de-sync event do we just need to grab the same logs as we provided above?

[Joseph Phillips (manadart)]

Looking through the initial logs supplied, I can see on machine 2 that we were spewing 404s around 25/5.
https://pastebin.canonical.com/p/3ZkgCDRpCt/

It almost looks as though the HTTP server worker was down, then managed to come up. This *might* explain inconsistencies.

[Joseph Phillips (manadart)]

Then there were a great deal of transaction issues:
https://pastebin.canonical.com/p/HD36KQxrWv/

[Joseph Phillips (manadart)]

Connectivity issues continue for some time with connection resets and broken pipes, then the lease issues that inevitably follow along with model leadership issues.

[Joseph Phillips (manadart)]

Based on the latest logs and their corresponding engine reports, I'm not seeing an obvious cause for inconsistencies.

Can we get some detail on what the alerts indicate when they are raised?

Errors are mostly the usual firewaller worker ones for ipv6-icmp, which will be fixed by upgrading. This should be done in the short term, because it will reduce load and log spam.

I can see a sudden jump up in load on ubuntu/2 here:
https://grafana.admin.canonical.com/d/sR1-JkYmz/juju2-controllers-thumpers?orgId=1&var-controller=prodstack5-prodstack5-prodstack-is&var-host=All&var-node=All&from=1691466766995&to=1691556622995

There's nothing to explain this, and it is obviously affecting API response times.

[Romain Couturat (romaincout)]

around 2023-10-06T13:56:52+00:00 , running
alias jsft='juju status --format tabular'
4 times in a row gave me 3 different results (look at landscape-client scale) https://pastebin.canonical.com/p/QCGgK4kf24/

[Joseph Phillips (manadart)]

When you're seeing this, please capture the output with --debug supplied to the status command.

That will give us some idea of which controller we're actually connecting to for the result.