Comment 6 for bug 416990

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

...

>> Getting:
>> - the branch being operated on
>> - how many packs and indices the branch had, its format
>
> Here's an example (taking the first process listing in that top output
> as an example):
>
> https://pastebin.canonical.com/21515/

So that tells us:

1) It is a mysql maria branch, presumably: ~maria-captains/maria/1.5
2) It is a 1.9 format branch (so not --2a)
3) It currently has open one of the very large text index files:

python2.4 13024 codehost 4r REG 254,0 14913650 97550603
   /srv.../indices/05ff4b81be41c0ce966dc98a45e87208.tix
python2.4 13024 codehost 12r FIFO 0,5 325267728 pipe

But that seems to be the only file open as part of the actual branch.
All the rest appear to just be python eggs or source, or extensions, etc.

4) You are running on python2.4

I know of at least one fairly serious memory related fix in python 2.5:
http://evanjones.ca/python-memory-part3.html

Whether that is at play here or not, I don't know. But long-lived
processes and python2.4 don't play nicely with memory.

5) Individually no process seems to be more that 7-800MB, but since you
have several running at once, it adds up.

6) The process has been running for ~6min. My guess is that it is
someone doing a full checkout (from scratch) of the code.

7) 28046 codehost 20 0 410m 96m 2300 R 44 1.2 293:03.42
/usr/bin/python2.4
/srv/bazaar.launchpad.net/production/launchpad/bin/twistd --pidfile=/srv/baz

The service process that is spawning all of this has a surprising (to
me) amount of memory consumed. I honestly don't know what it is doing,
but it has allocated 410MB of ram. (If I understand VIRT correctly.)
96MB resident is reasonable, though.

8) 542MB resident == 6.8% ram, so you have approx 8GB of ram on the
system. So you can run <20 of these processes before having very serious
issues. (And probably more like <10 before you are no longer caching
disk buffers, and starting to have problems.) (I guess that was also in
the meminfo at the beginning.)

9) I assume 'bzr lp-serve' is used to send/receive data from the outside
world, not for the mirroring process?

I'll try a little bit of experimenting here, to see if I can pinpoint
the memory consumption a little bit better. It certainly would help if
we knew what actual action is being performed...

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkqVmCsACgkQJdeBCYSNAAPRawCdEdgYGU7QfxpRrDQ2RtyOrOE6
hfcAn2Vp3We4SmZ6q3LmEdNLTDpWC6kM
=g2sQ
-----END PGP SIGNATURE-----