Apache mod_dav_svn leaking memory

[Lachlan]

Binary package hint: libapache2-svn

I have a large Subversion repository hosted with read access over plain HTTP. When checking out from the repository, the Apache process that is serving the WebDAV REPORT request starts consuming more and more memory until all memory and swap space is consumed and the machine becomes inaccessible. The only work-around I have found is to set "ulimit -v 3000000" before launching Apache, which effectively kills the process when virtual memory usage hits 3Gb. The client can then restart the checkout and eventually it will be completed.

I have only noticed the leak when using Subversion clients, and tests with Apache Bench don't seem to cause it, so it would appear to be a problem with the Subversion module. I can't be absolutely certain at this stage though.

lsb_release -rd: Description: Ubuntu 8.04.1 Release: 8.04

uname -a: Linux x.y.com 2.6.24-19-server #1 SMP Wed Aug 20 18:43:06 UTC 2008 x86_64 GNU/Linux

libapache2-svn: 1.4.6dfsg1-2ubuntu1
apache2: 2.2.8-1ubuntu0.3

The machine has 4 CPUs and 4Gb of RAM. Running x86_64 Ubuntu.

[hyperial (denny-hyperial)]

I have encountered the same problem on test machine I have setup under Intrepid Ibex 32-bit BETA. My machine is smaller with 1gb RAM & Pentium D @ 2.45GHz.

Using TortoiseSVN, I get about 1300mb of data before Apache restarts, thus killing the checkout/update... This is especially bad for me since my repository is ~75gb.

[hyperial (denny-hyperial)]

In addition, here is the apache error...

[Tue Oct 07 18:32:53 2008] [notice] child pid 26268 exit signal Segmentation fault (11)

Attached is the visual error received in TortoiseSVN

[enlavin (miguel-hernandez)]

I'm running a debian etch server (4 CPUs, 4GB RAM, x86_64) with HTTPS, LDAP auth, mod_svn and I'm experiencing the same issues with large checkouts (~1.7GB). It's not ubuntu and I don't know if is appropriate to report here, but I think it's interesting just for the record.

apache2 (2.2.3-4+etch4)
apache2-mpm-prefork (2.2.3-4+etch4)
libapache2-svn (1.4.2dfsg1-2)

uname -a: Linux xeon 2.6.18-4-amd64 #1 SMP Mon Mar 26 11:36:53 CEST 2007 x86_64 GNU/Linux

I also think that this is a problem directly related to mod_svn.

[Ralf ter Veer (rtv-team-mediaportal)]

Same here on a virtual root server with 512 MB RAM...

apache2.2-common ( 2.2.8-1ubuntu0.3) - tested mpm-prefork and mpm-worker
libapache2-svn (1.5.1dfsg1-1ubuntu2~hardy2)
libsvn1 (1.5.1dfsg1-1ubuntu2~hardy2)

Checking out the full repository (around 8GB - mainly text and a few xml files) will certainly and reproducibly stall the machine to a halt because apache's memory usage climbs from initial ~12 MB to the machine's maximum.

uname -a: Linux xxx.providerbox.net 2.6.18-12-fza-686-bigmem #1 SMP Sun May 18 13:01:05 CEST 2008 i686 GNU/Linux

[Ralf ter Veer (rtv-team-mediaportal)]

I've found an interesting detail:
If I use the subversion commandline binaries 1.4.6 this does NOT happen.
The server only leaks memory if it is accessed by recent TortoiseSVN clients.

[Alen (phone-gr)]

Same here, when user commits using TortoiseSVN the machine does not release memory.

[depe (depe-ubuntu)]

I found the same issue using Ubuntu 9.04 32bits server
Linux hdm-s004 2.6.28-11-server #42-Ubuntu SMP Fri Apr 17 02:48:10 UTC 2009 i686 GNU/Linux
libapache2-svn 1.5.4dfsg1-1ubuntu2
subversion 1.5.4dfsg1-1ubuntu2

I also found some interesting information here: http://groups.google.se/group/linux.debian.bugs.dist/browse_thread/thread/d14c54c3b8a56255 and here http://www.sharp-tools.net/archives/000928.html

Issue seems to be memory leakage due to authentication requested for each path, which for big repos tree can be really problematic.

Workaround proposed is to add
SVNPathAuthz off
in /etc/apache2/mods-available/dav_svn.conf

>> I still need to test it again for a couple of huge svn checkout

From what I understood, this will cause mod_dav_svn stop sending authentication request for each path access.
Path authorization file still works correctly and users are still asked to enter user name / password the first time they connect

Still I'm very interested in a solution since this issues reduces authentication possibilities.

[depe (depe-ubuntu)]

I tried using SVNPathAuthz, but I still get memory being eaten up, so it's not really working

[hyperial (denny-hyperial)]

I ended up just using SVN commandline for large checkouts and commits and Tortoise in Windows for day to day smaller things and haven't had the problem since.

[depe (depe-ubuntu)]

I found another option which works correctly now:

I did set 'SVNAllowBulkUpdates Off' in my mod_dav_svn config file and now memory consumption stays normal (<450MB with about 10 users) during the whole checkout, even with about 1.6GB data / 6000 files to get (few hours required).
Before (default value is On when option is not explicitly set), I could easily get 500MB used after 15mn, if I let the process it would eventually fill up the whole RAM/swap.

To summarize, this option will force SVN clients to request only parts of the tree to get, and not the entire tree within one request.
Default behavior is to let the client decide which amount of data to get for one request.
This could explain why TortoiseSVN 1.6 has different behavior than SVN 1.6 command line version.
I indeed only got this memory problem using TortoiseSVN and not with SVN command line.

report to subversion team as http://subversion.tigris.org/issues/show_bug.cgi?id=3084

[hyperial (denny-hyperial)]

Thanks Depe, That setting seemed to fix my problem using Tortoise completely, I've encountered no problem since updating it. The memory use on my server actually decreased a little during a large checkout rather than running out of control. :-)

The processor load seems to be less as well.

[Maarten Bezemer (veger)]

Thanks for taking the time to report this bug in the upstream bug tracking system this is a tremendous help. Launchpad has the ability to watch lots of upstream bug trackers and this can be done by following the procedure documented at https://wiki.ubuntu.com/Bugs/Watches. I've added the bug watch for this bug report.

Unfortunately, according to the upstream report the issue is still not fixed...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.