squid runs out of file descriptors

[Paul Gear]

On one of our ubuntu-repository-cache units, squid reported the following errors in the log:

2016/03/08 06:50:21| Attempt to open socket for EUI retrieval failed: (24) Too many open files
2016/03/08 06:50:21| /srv/ubuntu-repository-cache/squid/04/31: (24) Too many open files
...
2016/03/08 06:54:04| Reserved FD adjusted from 100 to 64538 due to failures
...
2016/03/08 06:54:04| WARNING! Your cache is running out of filedescriptors

When the cache goes into this state, performance is severely limited, per the explanation at http://www.squid-cache.org/mail-archive/squid-users/201210/0223.html

Looking at the configuration, it seems that squid is configured to use a maximum of 65536 file descriptors, but the system limit is still 1024:

root@juju-UUID-machine-2:~# sudo -u proxy bash -c 'ulimit -n'
1024
root@juju-UUID-machine-2:~# squidclient -p 8000 mgr:info | grep 'file desc'
        Maximum number of file descriptors: 65536
        Largest file desc currently in use: 13
        Number of file desc currently in use: 13
        Available number of file descriptors: 65523
        Reserved number of file descriptors: 64538

ubuntu-repository-cache should correctly configure the system to allow squid to use all of its file descriptors.

[Paul Gear (paulgear)]

It should be noted that both /etc/init/squid3.conf and /etc/init.d/squid-deb-proxy include lines to raise the ulimit, but as the above shows, this does not seem to take effect correctly.

The instance in question was in GCE, if that makes any difference.

[Paul Gear (paulgear)]

Further to the above, the following can be used to confirm that the ulimit change is not effective:

root@juju-UUID-machine-2:~# grep open.files /proc/$(pidof squid3)/limits
Max open files 1024 4096 files

[Nick Moffitt (nick-moffitt)]

http://www.squid-cache.org/mail-archive/squid-users/199909/0374.html (admittedly from 1999, predating squid3) suggests this isn't actually a ulimit problem, but a reserved FD crisis.

 2016/03/08 06:54:04| Reserved FD adjusted from 100 to 64538 due to failures

This hints that it basically reserved nearly all of them. This can't be right.