When using Ubuntu Server 12.04 with or without the latest updates (kernels 3.2.0-23 and 3.2.0-29, x86_64) as an NFS server with fairly heavy reading activity from clients (no writing), from a volume with a lot of small files, split into many subdirectories (with about 5-10 files or subdirectories per directory, in a tree-like structure not unlike that of Squid proxy), available memory is quickly exhausted, however no single process shows that much memory being used, nor does the "buffers" or "cached" in "free" command output. The server eventually runs out of memory and crashes.
slabtop shows that majority of memory is being used by idr_layer_cache (3.6G on a sever with 4G of RAM shortly before the kernel started killing processes and eventually crashed).
The filesystem being shared is ext4. Clients (also the same version of Ubuntu Server) mount the volume in read-only mode, with default options.
P.S. Also tried i386 version, with the same result.