smbd spawns hundreds/thousands of processes

[Scott Alfter]

Sometime in the past week, my server has started spawning hundreds (or even thousands) of smbd processes, driving the load average through the roof. Once every minute or so, it appears to start a new one.

Output from ubuntu-bug is attached.

[Scott Alfter (scott-sender-82b1fa)]

I think the recent change that caused this problem was where I told my router (a WRT54GL running Tomato) to start writing logfiles to a Samba share. greyhole --logs reported that the logfile was being held open. I turned off logging, rebooted the router, stopped Samba, made sure all its processes were dead (and that lsof no longer reported the logfile as being held open), and restarted Samba. Where the smbd process count would've previously gone up at a steady pace with many of them trying to take 100% CPU, the process count is now staying relatively low (about equal to the number of shares it's serving up) and the load average is staying low (currently 0.00).

So now we know what's causing the problem and how to keep it from happening, but why is it a problem? Files held open by clients shouldn't cause this problem. Transmission, for instance, can keep downloaded files open for days or weeks at a time, but this only became a problem when I told my router to start writing logfiles.

[Scott Alfter (scott-sender-82b1fa)]

More info: the router logs bandwidth usage information to the same Samba share; it's done that for over a year and a half with no issues.

[Scott Alfter (scott-sender-82b1fa)]

...and it turns out the router is not to blame, as when I logged into my server this morning to check on it, top looked something like this:

top - 08:38:30 up 1 day, 14:40, 1 user, load average: 874.41, 870.45, 860.30
Tasks: 1050 total, 877 running, 173 sleeping, 0 stopped, 0 zombie
%Cpu(s): 13.9 us, 85.6 sy, 0.3 ni, 0.0 id, 0.0 wa, 0.2 hi, 0.0 si, 0.0 st
KiB Mem: 3491160 total, 3340508 used, 150652 free, 382332 buffers
KiB Swap: 2097148 total, 40056 used, 2057092 free. 206400 cached Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    8 root 20 0 0 0 0 S 4.6 0.0 93:24.75 rcuos/0
    9 root 20 0 0 0 0 S 4.6 0.0 107:11.97 rcuos/1
 5900 salfter 20 0 25740 2464 1132 R 1.3 0.1 0:00.27 top
 2716 salfter 20 0 1400368 85772 2872 S 1.0 2.5 10:05.09 sabnzbdplus
 1057 salfter 20 0 280604 5936 3928 R 0.7 0.2 0:50.99 smbd
 1940 salfter 20 0 280604 5960 3952 R 0.7 0.2 0:50.51 smbd
 2615 salfter 20 0 280004 5852 3868 R 0.7 0.2 2:07.74 smbd
 3422 salfter 20 0 280012 5856 3872 R 0.7 0.2 2:06.06 smbd
 4002 salfter 20 0 280012 5860 3876 R 0.7 0.2 2:03.86 smbd
 4219 salfter 20 0 280012 5864 3880 R 0.7 0.2 2:02.97 smbd
 4330 salfter 20 0 279652 5824 3840 R 0.7 0.2 10:41.66 smbd
 4553 salfter 20 0 281156 6000 3992 R 0.7 0.2 0:01.40 smbd
 4896 salfter 20 0 280012 5860 3876 R 0.7 0.2 2:00.07 smbd
 5229 salfter 20 0 280136 5864 3880 R 0.7 0.2 1:58.06 smbd
 5351 salfter 20 0 281156 5988 3980 R 0.7 0.2 0:00.46 smbd
 5449 salfter 20 0 279696 5844 3860 R 0.7 0.2 7:19.14 smbd
 5748 salfter 20 0 280604 5940 3932 R 0.7 0.2 0:43.70 smbd
 6203 salfter 20 0 280604 5964 3956 R 0.7 0.2 0:43.20 smbd
 6410 salfter 20 0 279652 5824 3840 R 0.7 0.2 6:35.06 smbd
 7511 salfter 20 0 279660 5828 3844 R 0.7 0.2 6:01.25 smbd
 8540 salfter 20 0 280180 5860 3876 R 0.7 0.2 1:50.95 smbd
 9098 salfter 20 0 280180 5892 3908 R 0.7 0.2 1:46.50 smbd
 9157 salfter 20 0 280604 5924 3916 R 0.7 0.2 0:38.41 smbd
10018 salfter 20 0 280180 5876 3892 R 0.7 0.2 1:43.70 smbd
10233 salfter 20 0 280180 5888 3904 R 0.7 0.2 1:42.71 smbd
10382 salfter 20 0 279660 5828 3844 R 0.7 0.2 4:54.70 smbd
12170 salfter 20 0 279676 5836 3852 R 0.7 0.2 4:35.70 smbd
12363 salfter 20 0 280180 5868 3884 R 0.7 0.2 1:38.48 smbd
13355 salfter 20 0 280180 5880 3896 R 0.7 0.2 1:33.46 smbd
13907 salfter 20 0 280180 5896 3912 R 0.7 0.2 1:32.28 smbd
15617 salfter 20 0 280708 5956 3952 R 0.7 0.2 0:27.87 smbd
16100 salfter 20 0 280708 5928 3924 R 0.7 0.2 0:27.61 smbd
16320 salfter 20 0 280736 5924 3920 R 0.7 0.2 0:27.66 smbd

smbd has spawned 902 processes.

[Scott Alfter (scott-sender-82b1fa)]

>More info: the router logs bandwidth usage information to the same Samba
>share; it's done that for over a year and a half with no issues.

Until now, apparently. I reconfigured the router to write its bandwidth logs to internal flash (had to back off on the write frequency to keep from killing the flash) and unmapped the Samba share it was using for that purpose. The Samba server's not needed any attention over the weekend, and when I logged in a few minutes ago, it reported its current load average as 0.01.