HA Juju controllers showing inconsistent status

Bug #2020849 reported by James Simpson
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Canonical Juju

Bug Description


We've experienced an issue with the consistency and stability of our Juju controllers, and are struggling to pinpoint what's actually happening.

We're operating a HA controller set, running Juju 2.9.42, deployed in an Openstack cloud.

Symptoms we've observed have been:

* Issues with the stability of relationship hooks in deployed models (we have observed issues with relationships being created, updated, and departed)

* Controllers returning inconsistent "juju status" results
  When running "juju status --debug" to make sure we get one result from each controller, we have observed that at least one controller will consistently return a different result than the other(s).
  For example, this paste shows both secondary controllers reporting the primary controller as "agent-lost", while the primary disagrees: https://pastebin.ubuntu.com/p/VHrwFbm79Z/

Controller logs from the period in question have been made available via secure portal https://juju-controller-reports.admin.canonical.com/ps5-prodstack-is/

Model logs for the specific model in which we observed relationship hook issues are located in "special-request" under that directory.

Please advise if there are any additional logs we should supply, any metrics we can gather from the time, or anything else.


Tags: canonical-is
Tom Haddon (mthaddon)
tags: added: canonical-is
Revision history for this message
Ian Booth (wallyworld) wrote :

The controllers share agent connectivity info (aka presence) using pubsub. I don't think there's an explicit delivery guarantee for such messages.

Logging to turn on would be


Logs could also contain messages matching the format string

"%p programming error, e.ch=%v did not accept %v - missing Unwatch?\nwatch source:\n%s"

Extra relation debug can be obtained by setting


Changed in juju:
status: New → Incomplete
Revision history for this message
James Simpson (jsimpso) wrote :

Thanks Ian - I've configured those via "juju model-config -m controller logging-config=...", next time we experience an de-sync event do we just need to grab the same logs as we provided above?

Junien Fridrick (axino)
Changed in juju:
status: Incomplete → New
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.