juju 2.6.3 is slow, mongodb queries take long to respond

Bug #1832294 reported by Przemyslaw Hausman on 2019-06-11
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
juju
High
Tim Penhey
2.6
High
Tim Penhey

Bug Description

On a new deployment 'juju status' takes around 2 minutes to show results. Load on one of the juju controller machine is 2x higher than available resources can provide.

Juju Controllers are deployed in HA to 3x KVMs with 8 CPUs, 24GB RAM and 100GB disk. System load on one of the juju controlles KVM is 2x higher than available resources can provide (load 19 on 8 CPUs): https://pastebin.ubuntu.com/p/sNhcj7Sd7y/

After ~10 hours since deployment of juju controllers, syslog file is ~740MB and contains mostly mongodb messages: https://pastebin.ubuntu.com/p/RsrM4sg2td/

Logs from all 3 juju controllers: https://drive.google.com/open?id=1b_bsE-4asyQI0J-gqcslWKMgKn-nJvJU

Joseph Phillips (manadart) wrote :

What provider was this set up on?

Can you provide the bundle that was deployed?

Joseph Phillips (manadart) wrote :

Many issue symptoms in the logs.

Among those are the same messages as appear in https://bugs.launchpad.net/juju/+bug/1825854

Joseph Phillips (manadart) wrote :

^ Red herring.

2019-06-10 17:59:30 WARNING juju.network.containerizer bridgepolicy.go:296 container "17/lxd/0" wants spaces "internal-space", "oam-space", but host machine "17" has "oam-space" missing "internal-space"

Looks legit. It is not the same as seen in #1825854

Changed in juju:
status: New → Triaged
Joseph Phillips (manadart) wrote :

The logs are showing a slew of "model not found" errors, which we have observed in the field just this week in the field and are attempting to diagnose.

Is it possible to get a DB dump from this controller?

I will mark this as incomplete pending more information.

Changed in juju:
status: Triaged → Incomplete
Tim Penhey (thumper) wrote :

In part at least fixed by https://github.com/juju/juju/pull/10317

Joseph Phillips (manadart) wrote :
Tim Penhey (thumper) on 2019-06-13
Changed in juju:
status: Incomplete → In Progress
importance: Undecided → High
assignee: nobody → Tim Penhey (thumper)
milestone: none → 2.6.4
Joseph Phillips (manadart) wrote :

Relevant patch for 2.6 is now:
https://github.com/juju/juju/pull/10322

Appropriate parts against 2.5 under:
https://github.com/juju/juju/pull/10321

Changed in juju:
status: In Progress → Fix Committed
Anastasia (anastasia-macmood) wrote :

Bigger merge PR (https://github.com/juju/juju/pull/10326) brought this change into develop.

Changed in juju:
milestone: 2.6.4 → 2.7-beta1
David Coronel (davecore) wrote :

I subscribed field-critical to this bug as this is affecting a customer deployment.

David Coronel (davecore) wrote :

Unsubscribed field-critical since there's a workaround to use Juju 2.5 and fix seems to be on the way.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers