possible memory leak in Hardy's squid

[MatthewMetzger]

Binary package hint: squid

Description: Ubuntu hardy (development branch)
Release: 8.04

apt-cache policy squid
squid:
  Installed: 2.6.18-1ubuntu3
  Candidate: 2.6.18-1ubuntu3
  Version table:
 *** 2.6.18-1ubuntu3 0
        500 http://us.archive.ubuntu.com hardy/main Packages
        100 /var/lib/dpkg/status

When I run squid, the memory (RAM) cache of the system slowly creeps upward until almost all available memory is in the cache. I have watched this using the output of "free -m". However, when I restart the server, and then immediately stop squid "/etc/init.d/squid stop", the memory cache does not grow out of control like when squid is running.

This is a problem because the cache doesn't seem to clear itself out and my server has crashed several times when other processes needed to use that memory. I'm getting out of memory errors.

I'm not totally sure if squid is causing this problem, but it seems most likely as the memory cache does not grow when squid service is stopped.

[MatthewMetzger (matthew-metzger)]

I have more news on this bug. It is possibly a bug with Hardy in general and not with squid specifically.

I turned off squid and apache and let the machine just run (doing almost nothing). The cache did not appear to grow much at all, definitely not at the predictable rate it did when squid was turned on. However, after 16 hours the cache had grown to consume most of the memory even when not much was really running. The activity of squid was probably just making the memory problem happen more quickly.

This install of Hardy server is running on the most recently available Mac mini hardware.

Here's the output of "free -m":

             total used free shared buffers cached
Mem: 979 915 64 0 69 778
-/+ buffers/cache: 67 912
Swap: 2870 0 2870

Here's the output of "ps aux"
<code>
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.1 2844 1692 ? Ss Apr11 0:01 /sbin/init
root 2 0.0 0.0 0 0 ? S< Apr11 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S< Apr11 0:00 [migration/0]
root 4 0.0 0.0 0 0 ? S< Apr11 0:00 [ksoftirqd/0]
root 5 0.0 0.0 0 0 ? S< Apr11 0:00 [watchdog/0]
root 6 0.0 0.0 0 0 ? S< Apr11 0:00 [migration/1]
root 7 0.0 0.0 0 0 ? S< Apr11 0:00 [ksoftirqd/1]
root 8 0.0 0.0 0 0 ? S< Apr11 0:00 [watchdog/1]
root 9 0.0 0.0 0 0 ? S< Apr11 0:00 [events/0]
root 10 0.0 0.0 0 0 ? S< Apr11 0:00 [events/1]
root 11 0.0 0.0 0 0 ? S< Apr11 0:00 [khelper]
root 46 0.0 0.0 0 0 ? S< Apr11 0:00 [kblockd/0]
root 47 0.0 0.0 0 0 ? S< Apr11 0:00 [kblockd/1]
root 50 0.0 0.0 0 0 ? S< Apr11 0:00 [kacpid]
root 51 0.0 0.0 0 0 ? S< Apr11 0:00 [kacpi_notify]
root 122 0.0 0.0 0 0 ? S< Apr11 0:00 [kseriod]
root 164 0.0 0.0 0 0 ? S Apr11 0:00 [pdflush]
root 165 0.0 0.0 0 0 ? S Apr11 0:00 [pdflush]
root 166 0.0 0.0 0 0 ? S< Apr11 0:00 [kswapd0]
root 208 0.0 0.0 0 0 ? S< Apr11 0:00 [aio/0]
root 209 0.0 0.0 0 0 ? S< Apr11 0:00 [aio/1]
root 1397 0.0 0.0 0 0 ? S< Apr11 0:00 [ksuspend_usbd]
root 1398 0.0 0.0 0 0 ? S< Apr11 0:00 [khubd]
root 1466 0.0 0.0 0 0 ? S< Apr11 0:00 [khpsbpkt]
root 1521 0.0 0.0 0 0 ? S< Apr11 0:00 [ata/0]
root 1523 0.0 0.0 0 0 ? S< Apr11 0:00 [ata/1]
root 1525 0.0 0.0 0 0 ? S< Apr11 0:00 [ata_aux]
root 2199 0.0 0.0 0 0 ? S< Apr11 0:00 [knodemgrd_0]
root 2208 0.0 0.0 0 0 ? S< Apr11 0:00 [scsi_eh_0]
root 2209...

[Nick Barcet (nijaba)]

We had a quick chat with Matthew on irc. It is possibly a normal behavior and asked him to see if is cache was still growing if mysql was unloaded. waiting for his reply.

[Dustin Kirkland  (kirkland)]

Discussed at length with Matthew on IRC. Additionally, I created a Hardy-i386 virtual machine with 512MB of memory and was not able to reproduce the crashes or even memory leaks.

In summary, it's perfectly normal for all of memory to end up in the "cached" column of "free", as cached memory is essentially free memory, as the Linux kernel will quickly swap data out of cache as necessary [1].

Furthermore, the perceived "crashes" of the Hardy system may well have been the effects of the machine "swapping itself to death". My testing of Squid did not reveal any memory leaks. And Matthew seems to have chased down a problem in some other code:

<MatthewMetzger> a mysql query in Koha (the library software) that contains an "and" has the url formated as containing "+and+". This seemed to cause a loop.

Thanks,
:-Dustin

[1] http://gentoo-wiki.com/FAQ_Linux_Memory_Management

[Adrian Moisey (adrianmoisey)]

I manage 2 small squid boxes. Attached is a munin graph of the ones memory usage

[Adrian Moisey (adrianmoisey)]

Attached is the second box