HA Juju controllers showing inconsistent status

[James Simpson]

Hi,

We've experienced an issue with the consistency and stability of our Juju controllers, and are struggling to pinpoint what's actually happening.

We're operating a HA controller set, running Juju 2.9.42, deployed in an Openstack cloud.

Symptoms we've observed have been:

* Issues with the stability of relationship hooks in deployed models (we have observed issues with relationships being created, updated, and departed)

* Controllers returning inconsistent "juju status" results
  When running "juju status --debug" to make sure we get one result from each controller, we have observed that at least one controller will consistently return a different result than the other(s).
  For example, this paste shows both secondary controllers reporting the primary controller as "agent-lost", while the primary disagrees: https://pastebin.ubuntu.com/p/VHrwFbm79Z/

Controller logs from the period in question have been made available via secure portal https://juju-controller-reports.admin.canonical.com/ps5-prodstack-is/

Model logs for the specific model in which we observed relationship hook issues are located in "special-request" under that directory.

Please advise if there are any additional logs we should supply, any metrics we can gather from the time, or anything else.

Thanks!

## ADDED ###

A customer(using 2.9.49, 2,9,42) faced the same issue. and I also see this intermittently(not easy to reproduce though) when one of controllers are restarted.

I analyzed a bit of juju code below.

https://pastebin.ubuntu.com/p/kZDTB7PDtZ/

it seems that Presence(connections..?) has different value (in this case 'down' for machine or 'agent lost' for unit) then DB info. and if c.Presence.AgentStatus has down, juju status shows us it is down.

It seems that when one of controllers are rebooted, that value has changed or remained down.

I'm not sure about 3.x but is there related patch for 3.x? and can we backport it if there is?

I keep analyzing code but I would appreciate if juju team can give me any advice.

Thanks a lot.

[Ian Booth (wallyworld)]

The controllers share agent connectivity info (aka presence) using pubsub. I don't think there's an explicit delivery guarantee for such messages.

Logging to turn on would be

juju.worker.pubsub=TRACE
juju.worker.presence=TRACE

Logs could also contain messages matching the format string

"%p programming error, e.ch=%v did not accept %v - missing Unwatch?\nwatch source:\n%s"

Extra relation debug can be obtained by setting

juju.worker.uniter.relation=TRACE

[James Simpson (jsimpso)]

Thanks Ian - I've configured those via "juju model-config -m controller logging-config=...", next time we experience an de-sync event do we just need to grab the same logs as we provided above?

[Joseph Phillips (manadart)]

Looking through the initial logs supplied, I can see on machine 2 that we were spewing 404s around 25/5.
https://pastebin.canonical.com/p/3ZkgCDRpCt/

It almost looks as though the HTTP server worker was down, then managed to come up. This *might* explain inconsistencies.

[Joseph Phillips (manadart)]

Then there were a great deal of transaction issues:
https://pastebin.canonical.com/p/HD36KQxrWv/

[Joseph Phillips (manadart)]

Connectivity issues continue for some time with connection resets and broken pipes, then the lease issues that inevitably follow along with model leadership issues.

[Joseph Phillips (manadart)]

Based on the latest logs and their corresponding engine reports, I'm not seeing an obvious cause for inconsistencies.

Can we get some detail on what the alerts indicate when they are raised?

Errors are mostly the usual firewaller worker ones for ipv6-icmp, which will be fixed by upgrading. This should be done in the short term, because it will reduce load and log spam.

I can see a sudden jump up in load on ubuntu/2 here:
https://grafana.admin.canonical.com/d/sR1-JkYmz/juju2-controllers-thumpers?orgId=1&var-controller=prodstack5-prodstack5-prodstack-is&var-host=All&var-node=All&from=1691466766995&to=1691556622995

There's nothing to explain this, and it is obviously affecting API response times.

[Romain Couturat (romaincout)]

around 2023-10-06T13:56:52+00:00 , running
alias jsft='juju status --format tabular'
4 times in a row gave me 3 different results (look at landscape-client scale) https://pastebin.canonical.com/p/QCGgK4kf24/

[Joseph Phillips (manadart)]

When you're seeing this, please capture the output with --debug supplied to the status command.

That will give us some idea of which controller we're actually connecting to for the result.

[Seyeong Kim (seyeongkim)]

Hello @manadart

Could you please check description I added once?

Thanks in advance.

[Seyeong Kim (seyeongkim)]

problematic controller FullStatus info
https://pastebin.ubuntu.com/p/2PM549bGhY/

non-problematic controller FullStatus info
https://pastebin.ubuntu.com/p/zKmX943VCM/

presence information doesn't seem to be updated.

[Seyeong Kim (seyeongkim)]

I guess this is because race condition.

When one controller is restarted, the other controller gets presence request and run presence response.

presence response run UpdateServer() which update current status in Presence.

but as you can see juju status when one controller is restarted, shortly it shows us down or agent lost and getting back to normal.

While doing that, if presense response runs quickly before the status settle down. then it update Presence as down or agent lost.

This is assumption so I would appreciate if someone in juju team can confirm this.

[Seyeong Kim (seyeongkim)]

So, restarting another controller again fixes the status.

[John A Meinel (jameinel)]

I would certainly try to upgrade from 2.9.42 to 2.9.49. Some of the relation issues that you are seeing are likely fixed in 2.9.45: https://bugs.launchpad.net/juju/+bug/2011277
or
https://bugs.launchpad.net/bugs/2022855

You can see the high level of fixes here:
https://juju.is/docs/juju/roadmap#juju-juju-29-45

That doesn't directly impact `juju status` reporting whether or not the agent is 'lost', but it potentially addresses other issues that you're seeing.

[Seyeong Kim (seyeongkim)]

@jameinel

Thanks for your info, I already tested 2.9.49. Unfortunately I still can see the issue.
reproducer is quite simple but takes time.
(restarting one of controllers again and again)
Thanks a lot.

[Seyeong Kim (seyeongkim)]

3.1.5 has the same issue(3.1.0 doesn't work at deployment stage)
testing later version.