juju 2.6.3 is slow, mongodb queries take long to respond

[Przemyslaw Hausman]

On a new deployment 'juju status' takes around 2 minutes to show results. Load on one of the juju controller machine is 2x higher than available resources can provide.

Juju Controllers are deployed in HA to 3x KVMs with 8 CPUs, 24GB RAM and 100GB disk. System load on one of the juju controlles KVM is 2x higher than available resources can provide (load 19 on 8 CPUs): https://pastebin.ubuntu.com/p/sNhcj7Sd7y/

After ~10 hours since deployment of juju controllers, syslog file is ~740MB and contains mostly mongodb messages: https://pastebin.ubuntu.com/p/RsrM4sg2td/

Logs from all 3 juju controllers: https://drive.google.com/open?id=1b_bsE-4asyQI0J-gqcslWKMgKn-nJvJU

[Joseph Phillips (manadart)]

What provider was this set up on?

Can you provide the bundle that was deployed?

[Joseph Phillips (manadart)]

Many issue symptoms in the logs.

Among those are the same messages as appear in https://bugs.launchpad.net/juju/+bug/1825854

[Joseph Phillips (manadart)]

^ Red herring.

2019-06-10 17:59:30 WARNING juju.network.containerizer bridgepolicy.go:296 container "17/lxd/0" wants spaces "internal-space", "oam-space", but host machine "17" has "oam-space" missing "internal-space"

Looks legit. It is not the same as seen in #1825854

[Joseph Phillips (manadart)]

The logs are showing a slew of "model not found" errors, which we have observed in the field just this week in the field and are attempting to diagnose.

Is it possible to get a DB dump from this controller?

I will mark this as incomplete pending more information.

[Tim Penhey (thumper)]

In part at least fixed by https://github.com/juju/juju/pull/10317

[Joseph Phillips (manadart)]

Proposed another part:
https://github.com/juju/juju/pull/10319

[Joseph Phillips (manadart)]

Relevant patch for 2.6 is now:
https://github.com/juju/juju/pull/10322

Appropriate parts against 2.5 under:
https://github.com/juju/juju/pull/10321

[Anastasia (anastasia-macmood)]

Bigger merge PR (https://github.com/juju/juju/pull/10326) brought this change into develop.

[David Coronel (davecore)]

I subscribed field-critical to this bug as this is affecting a customer deployment.

[David Coronel (davecore)]

Unsubscribed field-critical since there's a workaround to use Juju 2.5 and fix seems to be on the way.