[SRU] samba "Too many files are currently in use."

[Frank]

Binary package hint: samba

When copying a directory with a good amount of files (> 1,000) to a Samba share from a Windows 7 client, I get the error "Too many files are currently in use.". This only happens on Ubuntu Server 9.10 (Samba 3.4.0) . I have tested on Ubuntu 9.04 (Samba 3.2.0) and the copy was successful.

While copying the files, I ran watch 'sudo lsof | grep smb | wc -l' and noticed that whenever the number of files open reaches somewhere between 1,200 and 1,300 I receive the error. If I wait a couple of seconds, the number of files open drop down significantly and I am able to choose "Try Again" on the error dialog, but once the number of files open climbs back up, the error reappears.

I am filling this as a bug because the copying of files works on a fresh install of 9.04, but fails on 9.10.

[Chuck Short (zulcss)]

Hi,

Are you getting a Windows dialog when you are doing the copying?

Regards
chuck

[Frank (fterragna)]

Yes, I am.

The dialog says "Too many files currently in use." and I am able to choose Try Again, Skip, or Cancel.

[Chuck Short (zulcss)]

In a terrminal what do you get on your samba server when you run the following command:

sysctl fs.file-max

Thanks
chuck

[Kai Giebeler (kai-giebeler)]

I receive the same problem:
Copying a few thousand files from a Samba-share (Ubuntu 9.10, Samba 3.4.0) to Windows 7-Client (Ultimate x64) produces the error message mentioned above after a few seconds. It's a Windows Dialog. The exact Message is:
"Too many Files are currently in use. Quit one or more programs, and then try again."
I can choose between [Try again], [Skip] and [Cancel]

Waiting a few seconds and then choosing [Try again] copies a few more files until the message reappears. the longer I wait the longer it copies without interruption.

Executing the command metioned above results in
"fs.file-max = 196754"
That's 10 times bigger than the total amount of files to be copied.

One thing I noticed:
"sudo smbstatus"
shows only two open files while idling but about 1013 open files in the moment when the message appears. While the dialog is open, the list shrinks continuously. After a few seconds it's nearly empty again.

I didn't notice the problem on Ubuntu 9.04 using Windows XP Professional 32-Bit as client.

A simple manual copy using the Windows GUI is basically possible, but other tools consuming a big amount of shared files seem to be unable to handle the exception properly (e.g. 7-zip crashes completely).

It seems that Samba doesn't close opened files fast enough. I cannot decide whether Windows forgets to close the files properly or Samba ignores the close event.

An (obfuscated) example line of an opened file in smbstatus:
"1920 1000 DENY_WRITE 0x20089 RDONLY EXCLUSIVE+BATCH /home/me My Files/my file.txt Wed Nov 4 21:42:44 2009"

It's a big problem for me since I'm totally unable to work with projects persisting of 1000+ files.
The problem can be permanently reproduced in my case (1 GB/s LAN). Network load is around 1 % except for some 20 % peaks. The test project consists of many small text files (source code).

If there's any more information I can provide to solve this problem, please let me know.

Kind regards

Kai

[Frank (fterragna)]

Kai,

That's *exactly* what happens to me. Couldn't be more clear of a description.

[Kai Giebeler (kai-giebeler)]

I just increased the amount of simultaneously opened files by samba inserting the following line into the [global]-section of smb.conf.
"max open files = 10240"
After a restart I was able to copy the above mentioned project without an issue - but that's rather a workaround than a solution. The files should be properly closed.

[Chuck Short (zulcss)]

Can you provide the information at the following:

http://wiki.samba.org/index.php/Capture_Packets

Regards
chuck

[robled (robled)]

I'd like to add a "me too" running Win7 and Ubuntu 9.10. It's consistently reproducible.

[Chuck Short (zulcss)]

@NTolerance:

It would help alot if you could attach a packet capture as descirbed in:

http://wiki.samba.org/index.php/Capture_Packets

Thanks
chuck

[Chuck Short (zulcss)]

Ok I was able to reproduce this, silly windows. I have a tcpdump and will be forwarding it off to them shortly.

Regards
chuck

[EugeneZ (eugene-zar)]

Also having the same problem described above. Attaching two wireshark pcap files from the Win7 computer. I was copying files from an SMB server when the problem occurred. During the wireshark capture I just let it run and when the error dialog appears I click Try Again over and over until Samba frees enough files to allow the transfer to continue, then a few seconds later it runs into the problem again.

[EugeneZ (eugene-zar)]

Never mind, I wasn't thinking. Those files are too big for attachments. I just uploaded them to a web server, hope that's alright. Here they are:

http://www.eugenez.net/downloads/smb-cap2.rar
http://www.eugenez.net/downloads/smb-cap3.rar

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:3.4.3-2ubuntu2

---------------
samba (2:3.4.3-2ubuntu2) lucid; urgency=low

  * debian/rules, debian/samba.if-up: allow "NetworkManager" as a recognized address
    family... it's obviously /not/ an address family, but it's what gets
    sent when using NM, so we'll cope for now. (LP: #462169). Taken from karmic-proposed.
  * debian/control: Recommend keyutils for smbfs (LP: #493565)
  * debian/patches/series, debian/patches/fix-too-many-open-files.patch: Windows 7 needs
    the Windows server open file handle number of 16384 handles. (LP: #462172)
 -- Chuck Short <email address hidden> Tue, 19 Jan 2010 12:54:46 -0500

[Chuck Short (zulcss)]

Statement of Impact: Transfering several thousand files from Ubuntu to Windows 7 will fail with the "Too many open files" message resulting in the copying only half working.

How this has been addressed: This has been fixed in lucid and upstream samba. I have verified that this has been fixed myself. i have attached the patch which fixes this issue.

How to reproduce:

1. Install Windows 7 in a virtual machine
2. Install a Samba Share on ubuntu
3. Create many small files in a Windows folder try to copy it over to the samba share.

There should not be any regressions with this patch.

[John Dong (jdong)]

This patch looks reasonable to me and also compatibility with Win7 is definitely SRU-worthy for this minimally invasive fix.

ACKing on behalf of ubuntu-sru -- proceed to upload to -proposed.

[Martin Pitt (pitti)]

Accepted samba into karmic-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Besterino (besterino)]

Had the same problem described above (copying folder with 77k files from windows 7 64Bit to ubuntu 9.10 server ended quickly in error message "too many files open...").

Installed the -proposed package manually (sudo aptitude install samba/karmik-proposed). After samba restart copying same folder works without interruption (windows error message that is).

I do notice a performance drop though. Where before the installation of the samba -proposed package I could copy a 2GB file at approx. 75-80MB/s to the samba share, the speed is now down to 60-65MB/s.

[Besterino (besterino)]

One further observation (but don't know whether that is related to the new package): The transfer speed drops dramatically after a while and is now down to 3.6-5MB/s (as shown in the details of the windows-copy-window) after approx 40.000 files (mostly small html files) have been copied.

[Martin Pitt (pitti)]

Thank you for testing!

Besterino [2010-02-18 22:35 -0000]:
> One further observation (but don't know whether that is related to the
> new package)

So the slowdown is not a regression then?

Thanks, Martin

--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)

[Besterino (besterino)]

@Martin:

Don't know exactly what you mean with "regression". Before installation of the patch transfer speed of the 2GB file started at approx. 119MB/sec and quickly dropped to approx 75-80MB/sec (and stayed there until the file is copied). With the patch copying of the 2GB file still starts at approx. 119MB/s and then quickly drops to 60-65MB/sec.

With a large number of rather small files (thousands of html-files around 55kb), the speed goes down at some point to 3.6-5MB/sec after a while. Did not pay attention to the exact time when, but I assume speed goes down significantly when the open file limit is reached.

Read speeds with the 2GB file did not change (approx 50-55MB/sec - which I consider a bit weird since lower than write speeds, but that's probably a different story).

Some background info: I'm running the smb-share on a raid5 consisting of 4 consumer 1TB samsung HDD (spinning at 5400rpm). The raid is exclusively used for hosting files for samba (and the FTP-server, which was of course not used during testing).

Unfortunately I am away over the weekend but will do some more testing next week. Happy to test different settings to narrow it down - just let me know what I should try.

[Martin Pitt (pitti)]

> Don't know exactly what you mean with "regression"

A regression is a bad behaviour which was introduced with this update and hasn't been present before.

The data transfer rates you mention are pretty similar, though. They could also be influenced by other traffic in the network, other running processes, etc.

If the update still works for you and addresses this bug, it looks okay to me.

[Besterino (besterino)]

 I agree - after further testing I am back at the old transfer speeds, so most likely not related to the update.

[Launchpad Janitor (janitor)]

This bug was fixed in the package samba - 2:3.4.0-3ubuntu5.5

---------------
samba (2:3.4.0-3ubuntu5.5) karmic-proposed; urgency=low

  * debian/patches/fix-libsmb-keepalive-packets.patch: Fix winbind authentication
    due to an invalid NTML2 signature. (LP: #479955)
    (https://bugzilla.samba.org/show_bug.cgi?id=6646)
  * debian/patches/fix-samba-point-and-print.patch: Allow automatic download
    of printer drivers from a Samba PDC (LP: #500457)
    (https://bugzilla.samba.org/show_bug.cgi?id=6568)
  * debian/patches/fix-too-many-openfiles.patch: When connecting to a Windows
    7 share users will get an error message "Too many open files are in use"
    after a certain number of files are copied. (LP: #462172 )
    (https://bugzilla.samba.org/show_bug.cgi?id=6837)
  * debian/patches/fix-win98-failed-connect.patch: Allow win98 clients to
    connect a samba server. Users will get an "Error 66" message. (LP: #502878)
    (https://bugzilla.samba.org/show_bug.cgi?id=6551)
 -- Chuck Short <email address hidden> Fri, 05 Feb 2010 15:03:50 -0500