Add MemoryMax systemd option to mailman3-web service

[Loïc Gomez]

Sometimes mailman3-web service uses too much memory. There's a probable memory leak, we got up to 4.5GB while it usually takes about 300MB:
https://pastebin.canonical.com/p/TtfKw978MH/ (sorry, private Canonical link).

We should add a systemd override to mailman3-web.service consisting of:
[Service]
MemoryMax=1G
Restart=on-failure

Restart=on-failure is already the default, but let's add it to make sure the service gets restarted, were it to change in the future.

For this option to work, we should also update systemd configuration /etc/systemd/system.conf (+ systemctl daemon-reload) with:
DefaultMemoryAccounting=yes

[Loïc Gomez (kotodama)]

We might also want to hardfail in case it's unable to properly restart under the limit with:
[Unit]
StartLimitIntervalSec=3m
StartLimitBurst=10

Thomas suggested we could also create a mailman systemd slice and add each service into it with limits so only those resources are affected, and have the cron job to run with systemd-run or update the charm to use systemd timers.

[Haw Loeung (hloeung)]

Is it really the mailman3-web service that's using too much memory on these units?

What I've found is that usually it's the regular daily/weekly/monthly update jobs consuming most of the memory.

Sure, doesn't hurt to add these limits in place. FWIW, this was recently bumped from 1G to 4G so we should take that into consideration when adding these limits. Perhaps make it tunable via juju charm config options.

[Loïc Gomez (kotodama)]

Yes, I had a incident where mailman3-web used too much memory, to the point where the daily cron job was unable to run for long and some essential processes were OOM killed.

I thought 1GB was enough for a web process, as on the other unit each process was using less than 300MB, but then it seems mailman3 allows for very demanding requests... Sorry about that.