Here's what I've found:
| $ while date; do ps afxuw | grep -E "[[:space:]]$(status jujud-machine-0 | awk '{ print $4 }')|RS\\S"; sudo pmap -XX $(status jujud-machine-0 | awk '{ print $4 }') | grep -E 'Rss|[[:space:]][1-9][[:digit:]]{5}'; sleep 600; done | Wed Nov 30 23:51:47 UTC 2016 | USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND | root 13713 29.0 25.7 1156072 528852 ? Ssl 22:03 31:35 /var/lib/juju/tools/machine-0/jujud machine --data-dir /var/lib/juju --machine-id 0 --debug | Address Perm Offset Device Inode Size Rss Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Swap KernelPageSize MMUPageSize Locked VmFlagsMapping | 034a6000 rw-p 030a6000 fd:01 1071550 208 112 95 20 0 4 88 112 88 0 0 4 4 0 rd wr mr mw me dw ac sd jujud | c81f158000 rw-p 00000000 00:00 0 242336 241052 241052 0 0 0 241052 239684 241052 6144 0 4 4 0 rd wr mr mw me ac sd [stack:13716] | c82ea00000 rw-p 00000000 00:00 0 141312 141256 141256 0 0 0 141256 140672 141256 47104 0 4 4 0 rd wr mr mw me ac sd | 1156076 528908 523262 7008 0 10608 511292 526164 511292 59392 0 392 392 0 KB | ... | Thu Dec 1 01:31:48 UTC 2016 | USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND | root 13713 29.9 44.9 1593580 922328 ? Ssl Nov30 62:23 /var/lib/juju/tools/machine-0/jujud machine --data-dir /var/lib/juju --machine-id 0 --debug | Address Perm Offset Device Inode Size Rss Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Swap KernelPageSize MMUPageSize Locked VmFlagsMapping | 034a6000 rw-p 030a6000 fd:01 1071550 208 108 95 16 0 4 88 108 88 0 0 4 4 0 rd wr mr mw me dw ac sd jujud | c81e49a000 rw-p 00000000 00:00 0 255384 254112 254112 0 0 0 254112 251616 254112 0 0 4 4 0 rd wr mr mw me ac sd [stack:13716] | c838000000 rw-p 00000000 00:00 0 223232 223232 223232 0 0 0 223232 218328 223232 0 0 4 4 0 rd wr mr mw me ac sd | c846a10000 rw-p 00000000 00:00 0 264832 227964 227964 0 0 0 227964 227264 227964 0 0 4 4 0 rd wr mr mw me ac sd | 1593584 922828 917828 6172 0 4340 912316 907352 912316 2048 0 444 444 0 KB
Full output - http://pastebin.ubuntu.com/23560803/
| $ while date; do echo "top10" | go tool pprof 'http://localhost:10000/debug/pprof/heap'; echo ""; sleep 600; done | Thu Dec 1 11:06:40 AEDT 2016 | Fetching profile from http://localhost:10000/debug/pprof/heap | Saved profile in /home/hloeung/pprof/pprof.localhost:10000.inuse_objects.inuse_space.008.pb.gz | Entering interactive mode (type "help" for commands) | (pprof) 270.67MB of 285.08MB total (94.95%) | Dropped 707 nodes (cum <= 1.43MB) | Showing top 10 nodes out of 71 (cum >= 2.50MB) | flat flat% sum% cum cum% | 86.93MB 30.49% 30.49% 104.93MB 36.81% github.com/juju/juju/apiserver.(*apiRoot).FindMethod.func1 | 68.01MB 23.86% 54.35% 68.01MB 23.86% launchpad.net/tomb.(*Tomb).init | 65.51MB 22.98% 77.33% 65.51MB 22.98% reflect.Value.call | 17MB 5.96% 83.29% 17MB 5.96% github.com/juju/juju/apiserver.newNotifyWatcher | 10.73MB 3.76% 87.05% 10.73MB 3.76% crypto/tls.(*block).reserve | 9MB 3.16% 90.21% 9MB 3.16% encoding/json.(*decodeState).literalStore | 7.50MB 2.63% 92.84% 7.50MB 2.63% runtime.malg | 3MB 1.05% 93.89% 3MB 1.05% encoding/json.stateBeginValue | 1.50MB 0.53% 94.42% 1.50MB 0.53% runtime.deferproc.func1 | 1.50MB 0.53% 94.95% 2.50MB 0.88% github.com/juju/juju/state.(*RelationScopeWatcher).loop | ... | Thu Dec 1 12:37:06 AEDT 2016 | Fetching profile from http://localhost:10000/debug/pprof/heap | Saved profile in /home/hloeung/pprof/pprof.localhost:10000.inuse_objects.inuse_space.017.pb.gz | Entering interactive mode (type "help" for commands) | (pprof) 541.86MB of 557.76MB total (97.15%) | Dropped 765 nodes (cum <= 2.79MB) | Showing top 10 nodes out of 46 (cum >= 11.81MB) | flat flat% sum% cum cum% | 224.49MB 40.25% 40.25% 254.49MB 45.63% github.com/juju/juju/apiserver.(*apiRoot).FindMethod.func1 | 125.51MB 22.50% 62.75% 125.51MB 22.50% launchpad.net/tomb.(*Tomb).init | 115.51MB 20.71% 83.46% 116.01MB 20.80% reflect.Value.call | 29MB 5.20% 88.66% 29MB 5.20% github.com/juju/juju/apiserver.newNotifyWatcher | 18.50MB 3.32% 91.98% 18.50MB 3.32% encoding/json.(*decodeState).literalStore | 14.84MB 2.66% 94.64% 14.84MB 2.66% crypto/tls.(*block).reserve | 7.50MB 1.35% 95.98% 7.50MB 1.35% runtime.malg | 5MB 0.9% 96.88% 5MB 0.9% encoding/json.stateBeginValue | 1MB 0.18% 97.06% 126.51MB 22.68% github.com/juju/juju/state.(*docWatcher).loop | 0.50MB 0.09% 97.15% 11.81MB 2.12% gopkg.in/mgo%2ev2.(*mongoSocket).readLoop
Again, full output - http://pastebin.ubuntu.com/23560801/
Here's what I've found:
| $ while date; do ps afxuw | grep -E "[[:space: ]]$(status jujud-machine-0 | awk '{ print $4 }')|RS\\S"; sudo pmap -XX $(status jujud-machine-0 | awk '{ print $4 }') | grep -E 'Rss|[[ :space: ]][1-9] [[:digit: ]]{5}'; sleep 600; done juju/tools/ machine- 0/jujud machine --data-dir /var/lib/juju --machine-id 0 --debug juju/tools/ machine- 0/jujud machine --data-dir /var/lib/juju --machine-id 0 --debug
| Wed Nov 30 23:51:47 UTC 2016
| USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
| root 13713 29.0 25.7 1156072 528852 ? Ssl 22:03 31:35 /var/lib/
| Address Perm Offset Device Inode Size Rss Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Swap KernelPageSize MMUPageSize Locked VmFlagsMapping
| 034a6000 rw-p 030a6000 fd:01 1071550 208 112 95 20 0 4 88 112 88 0 0 4 4 0 rd wr mr mw me dw ac sd jujud
| c81f158000 rw-p 00000000 00:00 0 242336 241052 241052 0 0 0 241052 239684 241052 6144 0 4 4 0 rd wr mr mw me ac sd [stack:13716]
| c82ea00000 rw-p 00000000 00:00 0 141312 141256 141256 0 0 0 141256 140672 141256 47104 0 4 4 0 rd wr mr mw me ac sd
| 1156076 528908 523262 7008 0 10608 511292 526164 511292 59392 0 392 392 0 KB
| ...
| Thu Dec 1 01:31:48 UTC 2016
| USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
| root 13713 29.9 44.9 1593580 922328 ? Ssl Nov30 62:23 /var/lib/
| Address Perm Offset Device Inode Size Rss Pss Shared_Clean Shared_Dirty Private_Clean Private_Dirty Referenced Anonymous AnonHugePages Swap KernelPageSize MMUPageSize Locked VmFlagsMapping
| 034a6000 rw-p 030a6000 fd:01 1071550 208 108 95 16 0 4 88 108 88 0 0 4 4 0 rd wr mr mw me dw ac sd jujud
| c81e49a000 rw-p 00000000 00:00 0 255384 254112 254112 0 0 0 254112 251616 254112 0 0 4 4 0 rd wr mr mw me ac sd [stack:13716]
| c838000000 rw-p 00000000 00:00 0 223232 223232 223232 0 0 0 223232 218328 223232 0 0 4 4 0 rd wr mr mw me ac sd
| c846a10000 rw-p 00000000 00:00 0 264832 227964 227964 0 0 0 227964 227264 227964 0 0 4 4 0 rd wr mr mw me ac sd
| 1593584 922828 917828 6172 0 4340 912316 907352 912316 2048 0 444 444 0 KB
Full output - http:// pastebin. ubuntu. com/23560803/
| $ while date; do echo "top10" | go tool pprof 'http:// localhost: 10000/debug/ pprof/heap'; echo ""; sleep 600; done localhost: 10000/debug/ pprof/heap pprof/pprof. localhost: 10000.inuse_ objects. inuse_space. 008.pb. gz com/juju/ juju/apiserver. (*apiRoot) .FindMethod. func1 net/tomb. (*Tomb) .init com/juju/ juju/apiserver. newNotifyWatche r tls.(*block) .reserve json.(* decodeState) .literalStore json.stateBegin Value deferproc. func1 com/juju/ juju/state. (*RelationScope Watcher) .loop localhost: 10000/debug/ pprof/heap pprof/pprof. localhost: 10000.inuse_ objects. inuse_space. 017.pb. gz com/juju/ juju/apiserver. (*apiRoot) .FindMethod. func1 net/tomb. (*Tomb) .init com/juju/ juju/apiserver. newNotifyWatche r json.(* decodeState) .literalStore tls.(*block) .reserve json.stateBegin Value com/juju/ juju/state. (*docWatcher) .loop mgo%2ev2. (*mongoSocket) .readLoop
| Thu Dec 1 11:06:40 AEDT 2016
| Fetching profile from http://
| Saved profile in /home/hloeung/
| Entering interactive mode (type "help" for commands)
| (pprof) 270.67MB of 285.08MB total (94.95%)
| Dropped 707 nodes (cum <= 1.43MB)
| Showing top 10 nodes out of 71 (cum >= 2.50MB)
| flat flat% sum% cum cum%
| 86.93MB 30.49% 30.49% 104.93MB 36.81% github.
| 68.01MB 23.86% 54.35% 68.01MB 23.86% launchpad.
| 65.51MB 22.98% 77.33% 65.51MB 22.98% reflect.Value.call
| 17MB 5.96% 83.29% 17MB 5.96% github.
| 10.73MB 3.76% 87.05% 10.73MB 3.76% crypto/
| 9MB 3.16% 90.21% 9MB 3.16% encoding/
| 7.50MB 2.63% 92.84% 7.50MB 2.63% runtime.malg
| 3MB 1.05% 93.89% 3MB 1.05% encoding/
| 1.50MB 0.53% 94.42% 1.50MB 0.53% runtime.
| 1.50MB 0.53% 94.95% 2.50MB 0.88% github.
| ...
| Thu Dec 1 12:37:06 AEDT 2016
| Fetching profile from http://
| Saved profile in /home/hloeung/
| Entering interactive mode (type "help" for commands)
| (pprof) 541.86MB of 557.76MB total (97.15%)
| Dropped 765 nodes (cum <= 2.79MB)
| Showing top 10 nodes out of 46 (cum >= 11.81MB)
| flat flat% sum% cum cum%
| 224.49MB 40.25% 40.25% 254.49MB 45.63% github.
| 125.51MB 22.50% 62.75% 125.51MB 22.50% launchpad.
| 115.51MB 20.71% 83.46% 116.01MB 20.80% reflect.Value.call
| 29MB 5.20% 88.66% 29MB 5.20% github.
| 18.50MB 3.32% 91.98% 18.50MB 3.32% encoding/
| 14.84MB 2.66% 94.64% 14.84MB 2.66% crypto/
| 7.50MB 1.35% 95.98% 7.50MB 1.35% runtime.malg
| 5MB 0.9% 96.88% 5MB 0.9% encoding/
| 1MB 0.18% 97.06% 126.51MB 22.68% github.
| 0.50MB 0.09% 97.15% 11.81MB 2.12% gopkg.in/
Again, full output - http:// pastebin. ubuntu. com/23560801/