Comment 9 for bug 1655169

Update on existing 6 node deployments:
 1. External 2.1beta3 controller (beta4 deployment in a separate model)
    mongod has grown to 2.6GiB from 2.1GiB in the last 19 hours

 2. Internal 2.1beta4 controller services places on the bootstrap machine 0 in :controller model:
    - mongod has grown to 4.6GiB from 2.7GiB in the last 19 hour
    - no OOM events killing mongo yet, and memory is nearly at capacity
              total used free shared buff/cache available
Mem: 15G 14G 234M 431M 1.0G 359M
Swap: 4.0G 4.0G 0B

We wrote a script to sort mongo collections * databases based on storage size. The most significant mongo collections are juju log related.

juju.statuseshistory: 25.03 MiB (0%)
blobstore.blobstore.chunks: 33.65 MiB (1%)
local.oplog.rs: 1.00 GiB (35%)
logs.logs: 1.76 GiB (61%)

--- database size
admin 951.00 B
presence 737.90 kiB
blobstore 33.65 MiB
juju 45.33 MiB
local 1.00 GiB
logs 1.76 GiB

The simple script: http://paste.ubuntu.com/23783766/

I'm wondering if Landscape using the :controller bootstrap machine 0 is what leads to the eventual OOM because of wiredTiger cache grabbing 50% of the machine's total memory.

Since juju doesn't specify --wiredTigerCacheSizeGB option to mongod, mongo assumes machine 0 owns all physical memory and wiredTiger will grow quickly to 50% of the physical memory on the system [1].

Since landscape deploys lxcs to machine 0 in the controller model, memory is impacted by the number of applications running in the lxcs so mongo claiming 50% of the physical memory will quickly lead to saturating swap space as mongo gets too big.

References:
[1] From https://docs.mongodb.com/manual/faq/storage/
"""
Starting in 3.4, the WiredTiger internal cache, by default, will use the larger of either:

50% of RAM minus 1 GB, or
256 MB.

NOTE
The storage.wiredTiger.engineConfig.cacheSizeGB limits the size of the WiredTiger internal cache. The operating system will use the available free memory for filesystem cache, which allows the compressed MongoDB data files to stay in memory. In addition, the operating system will use any free RAM to buffer file system blocks and file system cache.

To accommodate the additional consumers of RAM, you may have to decrease WiredTiger internal cache size.
"""