Runaway memory allocation in jujud unit agent

[Casey Marshall]

I've been seeing runaway memory allocations in jujud with 1.26, but only occasionally and intermittently.

I added the heap profiler to jujud's main and configured it to write the profile on SIGUSR2. Just caught it in the act, finally. It's been tricky, because when it happens on my machine, it usually becomes really unstable (OOM killer, etc.)

$ go tool pprof jujud jujud.mprof401040987
Entering interactive mode (type "help" for commands)
(pprof) top10
3378.66MB of 3396.71MB total (99.47%)
Dropped 118 nodes (cum <= 16.98MB)
      flat flat% sum% cum cum%
 3220.73MB 94.82% 94.82% 3378.66MB 99.47% time.NewTimer
  157.93MB 4.65% 99.47% 157.93MB 4.65% time.startTimer
         0 0% 99.47% 3378.66MB 99.47% github.com/juju/juju/worker/uniter.updateStatusSignal
         0 0% 99.47% 3395.16MB 100% github.com/juju/juju/worker/uniter/remotestate.(*RemoteStateWatcher).loop
         0 0% 99.47% 3395.16MB 100% github.com/juju/juju/worker/uniter/remotestate.NewWatcher.func1
         0 0% 99.47% 3396.21MB 100% runtime.goexit
         0 0% 99.47% 3378.66MB 99.47% time.After

I'm too fried to read the uniter code now, it's pretty late. Attached the binary & prof file, in case that helps.

[Casey Marshall (cmars)]

FWIW I haven't seen this happen in a really long time. I think we can close/mark invalid unless it surfaces again.

[Launchpad Janitor (janitor)]

[Expired for juju-core because there has been no activity for 60 days.]

[Anastasia (anastasia-macmood)]

@Jorge
This bug was expired before you tagged it.
Have you seen this behavior more recently?
Do you have a reproducible scenario?
Logs?

[Casey Marshall (cmars)]

I haven't experienced this issue in a looong time, and when I did, it was in a really early 1.26/2.0 beta. I think it's Expired/Invalid without some kind of recent confirmation.