"bzr check" in packs format takes more than 2GB RAM

[GuilhemBichot]

I'm running "bzr check -v" on a copy of our central repository (which takes a disk space of 1.4GB, and has format 1.14).
With bzr 2.1.0rc1 on Linux 64 bit, the process is killed by Linux due to "out of memory" (according to dmesg); the machine has 2GB RAM and no other significant process than this bzr; I repeated this problem twice in a row. I had done a similar command 1.5 years ago, it had worked ok and took 30 hours.
With bzr.dev on Linux 64 bit (another machine), the process is currently running fine, consuming 7GB according to "top" (the machine has 12 GB RAM).
Such huge memory consumption isn't practical.

[GuilhemBichot (guilhem-bichot)]

I confirm that with 2a this doesn't happen: memory usage is then 900 MB. That's far less than 7GB, though 900MB is still quite a lot; I don't see well why so much memory is needed.
Anyway, I have the 1.14-repo check running now on a machine with 12GB RAM, for 24 hours already, and hope it will finish successfully.

[GuilhemBichot (guilhem-bichot)]

So I have more results.
On my machine with 12GB RAM 4-core Linux I have two identical 1.14 shared repositories ("identical" as in "copied with cp -a"), I did "check -v" on one copy with bzr.dev, and on another copy with bzr 1.15.1.
* 1.15.1:
- "checking revisions" (87000 of them): took 24 hours
- "checking text storage: calculating text parents" (418000 of them): took 24 hours
- during those two phases, RAM usage was <6%
- then a few hours were spent in checking branches, RAM went to 50%
- the command terminated normally, taking in total 51 hours.
* bzr.dev:
- I killed it, out of patience, after 117 hours
- RAM usage was consistently 50% over all this time
- I cannot tell what the progress bar was saying, because I had redirected stdout to a file, and this makes the progress bar invisible.
* Note that the check with bzr.dev had been started 66 hours before the check with 1.15.1, so it's not a problem of parallel operations loading the machine. And there were no other significant operations on the machine for the whole time.

This seems to surely be a bug introduced between 1.15.1 and bzr.dev :-)
I suspect that you could reproduce this problem by creating a shared repo in 1.14 format, putting into it this branch:
https://code.launchpad.net/~mysql/mysql-server/mysql-6.0-codebase-bugfixing
and then running "bzr check -v". The above branch has the bulk of revisions of my shared repository.

[Martin Pool (mbp)]

Thanks, narrowing it down like that is helpful.

--
Martin <http://launchpad.net/~mbp/>

[GuilhemBichot (guilhem-bichot)]

Quick experiments with different versions, on the same 1.14 repo as above:

bzr.dev, 1.18.1:
They first do a check of each branch (fast) then go into
 checking commit contents:inventories 0/2
After 11 mins RAM starts growing from 6% to 50% (i.e. 6GB...)
I do control-C and see that bzr is in:
> bzrlib/osutils.py(656)sha_strings()
-> map(s.update, strings)

1.15.1, 1.17:
Seem to do things differently:
No working tree found at specified location.
Checking repository at 'file:///home/mysql_src/from_bk_internal/guilhem/'.
[###############/ ] checking revision graph:Checking cached revision graph 53000/87653
then goes into a 24-hour-long
[\ ] checking revision 3/87653
where RAM stays <6%.

So possibly this RAM consumption is a bug introduced between 1.17 and 1.18.1. Now I'm letting this for you to look at :-)