| 2016-11-29 13:56:42 |
Jacek Nykis |
bug |
|
|
added bug |
| 2016-11-29 13:57:41 |
Jacek Nykis |
description |
We recently upgraded a few environments to 1.25.8 and we started experiencing problems on some of them.
What we know so far:
* the problems only affects bigger environments, 8-10 machines and bigger. Smaller environments look stable
* on the problematic environments jujud uses lots of memory on node 0, for example nearly 1GB RES on bootstrap node with 2GB RAM
* we see "lost" agents ocassionally. It's intermittent, sometimes environments are fine for hours
* occasionally hooks end up in error state, we see error like this in the logs:
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.worker.uniter.filter filter.go:137 watcher iteration error: Closed explicitly
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
juju version is 1.25.8, running on amd64 trusty guests. |
We recently upgraded a few environments to 1.25.8 and we started experiencing problems on some of them.
What we know so far:
* the problems only affects bigger environments, 8-10 machines and bigger. Smaller environments look stable
* on the problematic environments jujud uses lots of memory on node 0, for example nearly 1GB RES on bootstrap node with 2GB RAM
* we see "lost" agents ocassionally. It's intermittent, sometimes environments are fine for hours
* occasionally hooks end up in error state, we see error like this in the logs:
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.worker.uniter.filter filter.go:137 watcher iteration error: Closed explicitly
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
juju version is 1.25.8, running on amd64 trusty guests.
I uploaded logs from the bootstrap node here:
https://private-fileshare.canonical.com/~jacek/lp1645729.tgz |
|
| 2016-11-29 13:59:05 |
Tom Haddon |
bug |
|
|
added subscriber The Canonical Sysadmins |
| 2016-11-29 13:59:42 |
Uros Jovanovic |
juju-core: importance |
Undecided |
Critical |
|
| 2016-11-29 17:42:55 |
Jacek Nykis |
description |
We recently upgraded a few environments to 1.25.8 and we started experiencing problems on some of them.
What we know so far:
* the problems only affects bigger environments, 8-10 machines and bigger. Smaller environments look stable
* on the problematic environments jujud uses lots of memory on node 0, for example nearly 1GB RES on bootstrap node with 2GB RAM
* we see "lost" agents ocassionally. It's intermittent, sometimes environments are fine for hours
* occasionally hooks end up in error state, we see error like this in the logs:
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.worker.uniter.filter filter.go:137 watcher iteration error: Closed explicitly
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
juju version is 1.25.8, running on amd64 trusty guests.
I uploaded logs from the bootstrap node here:
https://private-fileshare.canonical.com/~jacek/lp1645729.tgz |
We recently upgraded a few environments from juju 1.25.6 to 1.25.8 and we started experiencing problems on some of them.
What we know so far:
* the problems only affects bigger environments, 8-10 machines and bigger. Smaller environments look stable
* on the problematic environments jujud uses lots of memory on node 0, for example nearly 1GB RES on bootstrap node with 2GB RAM
* we see "lost" agents ocassionally. It's intermittent, sometimes environments are fine for hours
* occasionally hooks end up in error state, we see error like this in the logs:
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.worker.uniter.filter filter.go:137 watcher iteration error: Closed explicitly
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
2016-11-29 09:30:29 ERROR juju.api.watcher watcher.go:84 error trying to stop watcher: connection is shut down
juju version is 1.25.8, running on amd64 trusty guests.
I uploaded logs from the bootstrap node here:
https://private-fileshare.canonical.com/~jacek/lp1645729.tgz |
|
| 2016-11-29 21:23:12 |
Anastasia |
juju-core: status |
New |
Triaged |
|
| 2016-11-29 21:23:15 |
Anastasia |
juju-core: milestone |
|
1.25.9 |
|
| 2016-11-29 21:57:46 |
Haw Loeung |
bug |
|
|
added subscriber Haw Loeung |
| 2016-11-29 21:57:53 |
Haw Loeung |
bug |
|
|
added subscriber Canonical WebOps |
| 2016-11-30 07:24:43 |
Junien F |
bug |
|
|
added subscriber Junien Fridrick |
| 2016-12-05 13:17:18 |
Andrew Wilkins |
juju-core: status |
Triaged |
In Progress |
|
| 2016-12-05 13:17:20 |
Andrew Wilkins |
juju-core: assignee |
|
Andrew Wilkins (axwalk) |
|
| 2016-12-05 13:18:13 |
Andrew Wilkins |
nominated for series |
|
juju-core/2.0 |
|
| 2016-12-05 13:18:13 |
Andrew Wilkins |
bug task added |
|
juju-core/2.0 |
|
| 2016-12-05 13:18:13 |
Andrew Wilkins |
nominated for series |
|
juju-core/1.25 |
|
| 2016-12-05 13:18:13 |
Andrew Wilkins |
bug task added |
|
juju-core/1.25 |
|
| 2016-12-05 13:18:22 |
Andrew Wilkins |
juju-core: milestone |
1.25.9 |
|
|
| 2016-12-05 13:18:25 |
Andrew Wilkins |
juju-core/1.25: milestone |
|
1.25.9 |
|
| 2016-12-05 13:18:33 |
Andrew Wilkins |
juju-core/1.25: status |
New |
In Progress |
|
| 2016-12-05 13:18:38 |
Andrew Wilkins |
juju-core/1.25: importance |
Undecided |
Critical |
|
| 2016-12-05 13:18:41 |
Andrew Wilkins |
juju-core: importance |
Critical |
High |
|
| 2016-12-05 13:18:44 |
Andrew Wilkins |
juju-core/1.25: assignee |
|
Andrew Wilkins (axwalk) |
|
| 2016-12-05 13:18:47 |
Andrew Wilkins |
juju-core/2.0: importance |
Undecided |
High |
|
| 2016-12-05 13:18:50 |
Andrew Wilkins |
juju-core/2.0: assignee |
|
Andrew Wilkins (axwalk) |
|
| 2016-12-05 13:18:57 |
Andrew Wilkins |
juju-core/2.0: status |
New |
Triaged |
|
| 2016-12-05 16:09:38 |
Andrew Wilkins |
juju-core/1.25: status |
In Progress |
Fix Committed |
|
| 2016-12-06 08:52:58 |
Andrew Wilkins |
juju-core: status |
In Progress |
Fix Committed |
|
| 2016-12-06 08:53:05 |
Andrew Wilkins |
juju-core/2.0: status |
Triaged |
Fix Committed |
|
| 2016-12-07 10:13:33 |
Anastasia |
juju-core: milestone |
|
1.25.9 |
|
| 2016-12-07 10:14:49 |
Anastasia |
bug task added |
|
juju |
|
| 2016-12-07 10:15:02 |
Anastasia |
nominated for series |
|
juju/2.1 |
|
| 2016-12-07 10:15:02 |
Anastasia |
bug task added |
|
juju/2.1 |
|
| 2016-12-07 10:15:08 |
Anastasia |
juju: status |
New |
Fix Committed |
|
| 2016-12-07 10:15:12 |
Anastasia |
juju: importance |
Undecided |
High |
|
| 2016-12-07 10:15:17 |
Anastasia |
juju: milestone |
|
2.0.3 |
|
| 2016-12-07 10:15:26 |
Anastasia |
juju: assignee |
|
Andrew Wilkins (axwalk) |
|
| 2016-12-07 10:15:31 |
Anastasia |
juju/2.1: status |
New |
Fix Committed |
|
| 2016-12-07 10:15:34 |
Anastasia |
juju/2.1: importance |
Undecided |
High |
|
| 2016-12-07 10:15:43 |
Anastasia |
juju/2.1: assignee |
|
Andrew Wilkins (axwalk) |
|
| 2016-12-07 10:15:46 |
Anastasia |
juju/2.1: milestone |
|
2.1-rc1 |
|
| 2016-12-07 10:15:50 |
Anastasia |
bug task deleted |
juju-core/2.0 |
|
|
| 2016-12-07 10:16:12 |
Anastasia |
juju-core: importance |
High |
Critical |
|
| 2016-12-07 14:09:23 |
Curtis Hovey |
juju-core: milestone |
1.25.9 |
|
|
| 2016-12-07 16:33:22 |
Curtis Hovey |
juju-core/1.25: status |
Fix Committed |
Fix Released |
|
| 2016-12-13 02:35:00 |
Paul Gear |
attachment added |
|
1 week memory graph; 1.25.9-proposed upgrade was approximately 24 hrs ago https://bugs.launchpad.net/juju-core/+bug/1645729/+attachment/4790983/+files/Screenshot%20from%202016-12-13%2012-30-35.png |
|
| 2016-12-13 02:35:41 |
Paul Gear |
attachment added |
|
network traffic graphs for last week https://bugs.launchpad.net/juju-core/+bug/1645729/+attachment/4790984/+files/Screenshot%20from%202016-12-13%2012-31-29.png |
|
| 2016-12-13 23:00:10 |
Anastasia |
juju/2.1: milestone |
2.1-rc1 |
2.1-beta3 |
|
| 2016-12-15 23:54:15 |
Curtis Hovey |
juju/2.1: status |
Fix Committed |
Fix Released |
|
| 2017-01-10 00:55:57 |
Anastasia |
juju-core: status |
Fix Committed |
Fix Released |
|
| 2017-01-10 01:54:24 |
Paul Gear |
attachment added |
|
network traffic graphs for affected production env https://bugs.launchpad.net/juju-core/+bug/1645729/+attachment/4802153/+files/Screenshot%20from%202017-01-10%2011-46-47.png |
|
| 2017-01-10 01:54:48 |
Paul Gear |
attachment added |
|
memory graphs for affected production env https://bugs.launchpad.net/juju-core/+bug/1645729/+attachment/4802154/+files/Screenshot%20from%202017-01-10%2011-45-51.png |
|
| 2017-01-10 07:07:24 |
Paul Gear |
tags |
|
canonical-is |
|
| 2017-02-08 22:19:04 |
Curtis Hovey |
juju: status |
Fix Committed |
Fix Released |
|
