Slow USB transfer for FAT32

[Angelos Vlassopoulos]

This bug is "spin off" of bug "file transfers on USB disk are very slow "

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/197762

I did some tests and found out that when writing to my USB flash disk the speed is quite low. I observed that the situation is improved (or rather resolved) if the flash disk is formated as ext3, so this can be a problem with fat32. I also have an external USB hard disk (NTFS formated) which seems to work in satisfactory way. This further supports the fact that the problem affects fat32 formatted disks.

I ran several tests according to https://help.ubuntu.com/community/DiskPerformance (and also some other tests). In all cases I made sure that the flash disk was detected as a high speed device (USB 2.0) and not as a full speed device (USB 1.1) using the lsusb command.

Since the problem is more obvious for very large files I let dd create a test file for about 5 minutes, just to be on the safe side.

The facts:

When I generate a load using dd for about 5 minutes the results from dd are:

FAT32 USB flash disk: 1617933312 bytes (1.6 GB) copied, 305.66 s, 5.3 MB/s
ext3 USB flash disk: 41091136 bytes (41 MB) copied, 3.85154 s, 10.7 MB/s
NTFS Hard disk: 4294967295 bytes (4.3 GB) copied, 290.644 s, 14.8 MB/s (maximum file size was reached in less than 5mins

The same USB flash disk performs roughly twice as fast on a Windows XP machine.

The USB flash disk specificstions state that it is able to write at 10MB/sec and read at 15MB/sec

The hardware
M/B --> ECS GF7050VT-M5 (nvidia 7050 chipset), with the latest BIOS
USB Flash Disk --> Crucial Gizmo! plus 4Gb (JDOH4GB-730)
Hard Disk --> Western Digital WD300 connected with no-name IDE-to-USB equipment.

The sofware
Kubuntu Jaunty 64-bit with all the latest updates installed

[Angelos Vlassopoulos (avlass)]

Here is dd wrote in each case, after the test was concluded:

For USB flash disk (fat32):

avlass@avlass-desktop:~/Desktop/Test$ dd if=/dev/zero of=/media/CRUCIAL/test-file bs=32
^C50560416+0 records in
50560416+0 records out
1617933312 bytes (1.6 GB) copied, 305.66 s, 5.3 MB/s

For USB flash disk (ext3):

avlass@avlass-desktop:~$ dd if=/dev/zero of=/media/disk/test-file bs=32
^C1284098+0 records in
1284098+0 records out
41091136 bytes (41 MB) copied, 3.85154 s, 10.7 MB/s

For USB Hard disk (NTFS)
avlass@avlass-desktop:~$ dd if=/dev/zero of=/media/disk/test-file bs=32
dd: writing `/media/disk/test-file': File too large
134217728+0 records in
134217727+0 records out
4294967295 bytes (4.3 GB) copied, 290.644 s, 14.8 MB/s

[Angelos Vlassopoulos (avlass)]

Attached is the dstat output for the

USB Flash Disk (fat 32)

[Angelos Vlassopoulos (avlass)]

Attached is the dstat output for the

USB Flash Disk (ext3)

[Angelos Vlassopoulos (avlass)]

Attached is the dstat output for the

USB Hard Disk (NTFS)

[Angelos Vlassopoulos (avlass)]

I ran a few more tests on another PC and got some very interesting results.

The PC has an ASUS P5KPL-VM motherboard based on the Intel G31 chipset.

The results were the same for Kubuntu jaunty 64-bit and Kubuntu Jaunty 32-bit.

So it's not a problem of the hardware (unless it's the USB flash drive itself) and it's not a problem of the 64-bit version.

I also used the pci=routeirq and the pci=noacpi options on boot with no effect.

BUT... the problem does not appear in Hardy or Intrepid. It only appears in jaunty.

All tests were conducted after clean installs and with no updates.

The following are average times as reported from dd:

Hardy
avlass@avlass-desktop:~$ dd if=/dev/zero of=/media/disk/test-file bs=32
^C32215401+0 records in
32215401+0 records out
1030892832 bytes (1.0 GB) copied, 79.614 s, 12.9 MB/s

Intrepid
avlass@avlass-intrepid32:~$ dd if=/dev/zero of=/media/disk/test-file bs=32
^C112570144+0 records in
112570144+0 records out
3602244608 bytes (3.6 GB) copied, 315.495 s, 11.4 MB/s

Jaunty
avlass@avlass-kubuntu64:~$ dd if=/dev/zero of=/media/disk/test-file bs=32
^C63138817+0 records in
63138817+0 records out
2020442144 bytes (2.0 GB) copied, 311.369 s, 6.5 MB/s

I also attach the output of dstat for the Intrepid test

[Angelos Vlassopoulos (avlass)]

I formatted an external hard disk as fat32 and ran a test. It went pretty well. Here's what dd wrote as a result

avlass@avlass-kubuntu64:~$ dd if=/dev/zero of=/media/EXTFAT32/test-file bs=32
dd: writing `/media/EXTFAT32/test-file': File too large
134217728+0 records in
134217727+0 records out
4294967295 bytes (4.3 GB) copied, 320.683 s, 13.4 MB/s
avlass@avlass-kubuntu64:~$

I guess the problem is not with fat32, but with fat32 and USB sticks specifically.

I also attach the relevant dstat output.

I will try to test the speed with some other USB flash disks, but I don't have another right now.

In the mean time, if someone thinks that performing a certain test will help draw further conclusions, please let me know.

The reason I am pursuing this so much is that I believe it is possible that this problem is related with other known (or unknown) issues that affect USB trasfer rates. So any help will be much appreciated.

[Angelos Vlassopoulos (avlass)]

I ran a test with a different USB Flash drive. The Corsair Voyagermini (4GB). It has a maximum write speed of 5MB/sec, which I understand it's quite common for flash drives. It did reach its maximum speed of 5Mb/sec.

The Crucial Gizmo Plus (that I tested initially) also performed as fast, but never reached its own maximum speed of 10Mb/sec, that achieves on Windows and Intrepid.

I am always willing to perform any tests you think would be helpful.

[Angelos Vlassopoulos (avlass)]

No change with pci-nomsi boot option

[Friedrich Graeter (graeter)]

I can confirm this bug on my machine. Using dd on a FAT32 vs. ext4, I got similar transfer times.

I have this problem in Jaunty and Intrepid (since an update - the initial version of Interpid doesn't show the problem).

[rom85 (rom85)]

I have the same problem also since 8.04 when I started using Ubuntu. I've tried different distro's, flash drives and even PCs that I could get access to at work (all on intel chipsets). it's rather impossible to use usb as it doesn't work normally and i am confused of such limitations.The worst is there is by now no solution and i can't see something is being done in current approach. There are many bug reports on the internet but no real bugfix, only some actions to try. None of them helped me, neither their combinations. this is the worst thing about linux that no one cares for real(( (so sad...

[Przemek K. (azrael)]

Also see bug #197762 which might be affecting some of the commenting users

[Friedrich Graeter (graeter)]

The bug still exists in karmic.

But it seems to be somehow hardware-related, since it happens only on my notebook, but not on my other computers (the interesting thing is, that the file transfer is only slow when using FAT32 with my notebook).

My notebook has the following USB controller:

00:1d.0 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #1 (rev 01)
00:1d.1 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #2 (rev 01)
00:1d.2 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #3 (rev 01)
00:1d.3 USB Controller: Intel Corporation 82801G (ICH7 Family) USB UHCI Controller #4 (rev 01)
00:1d.7 USB Controller: Intel Corporation 82801G (ICH7 Family) USB2 EHCI Controller (rev 01)

A PC, which has no problems at all, has the following controller:

00:04.0 USB Controller: nVidia Corporation GeForce 7100/nForce 630i USB (rev a1)
00:04.1 USB Controller: nVidia Corporation MCP73 [nForce 630i] USB 2.0 Controller (EHCI) (rev a1)

So perhaps, this problem is controller-related...

[Jeremy Foshee (jeremyfoshee)]

Hi Angelos,

This bug was reported a while ago and there hasn't been any activity in it recently. We were wondering if this is still an issue? Can you try with the latest development release of Ubuntu? ISO CD images are available from http://cdimage.ubuntu.com/releases/ .

If it remains an issue, could you run the following command from a Terminal (Applications->Accessories->Terminal). It will automatically gather and attach updated debug information to this report.

apport-collect -p linux 392089

Also, if you could test the latest upstream kernel available that would be great. It will allow additional upstream developers to examine the issue. Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Once you've tested the upstream kernel, please remove the 'needs-upstream-testing' tag. This can be done by clicking on the yellow pencil icon next to the tag located at the bottom of the bug description and deleting the 'needs-upstream-testing' text. Please let us know your results.

Thanks in advance.

    [This is an automated message. Apologies if it has reached you inappropriately; please just reply to this message indicating so.]

[Angelos Vlassopoulos (avlass)]

Architecture: amd64
ArecordDevices:
 **** List of CAPTURE Hardware Devices ****
 card 0: NVidia [HDA NVidia], device 0: VT1708B Analog [VT1708B Analog]
   Subdevices: 2/2
   Subdevice #0: subdevice #0
   Subdevice #1: subdevice #1
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: avlass 1814 F.... pulseaudio
                      avlass 1843 F.... kmix
                      avlass 19251 F.... kmix
CRDA: Error: [Errno 2] No such file or directory
Card0.Amixer.info:
 Card hw:0 'NVidia'/'HDA NVidia at 0xfebf8000 irq 22'
   Mixer name : 'VIA VT1708B 8-Ch'
   Components : 'HDA:1106e721,10192965,00100100'
   Controls : 26
   Simple ctrls : 14
DistroRelease: Ubuntu 9.10
HibernationDevice: RESUME=UUID=293a1ebb-7f43-4fef-8fd4-981bb5cec5f2
InstallationMedia: Kubuntu 9.10 "Karmic Koala" - Release amd64 (20091027)
IwConfig:
 lo no wireless extensions.

 eth1 no wireless extensions.
Lsusb:
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 001 Device 003: ID 0951:1607 Kingston Technology Data Traveler 2.0
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
MachineType: ECS GF7050VT-M5
NonfreeKernelModules: nvidia
Package: linux (not installed)
ProcCmdLine: BOOT_IMAGE=/boot/vmlinuz-2.6.31-21-generic root=UUID=d1ceed78-8a27-4558-80e2-a548c07bee72 ro quiet splash
ProcEnviron:
 SHELL=/bin/bash
 LANG=en_US.UTF-8
 LANGUAGE=
ProcVersionSignature: Ubuntu 2.6.31-21.59-generic
RelatedPackageVersions:
 linux-backports-modules-2.6.31-21-generic N/A
 linux-firmware 1.26
RfKill:

Uname: Linux 2.6.31-21-generic x86_64
UserGroups: adm admin cdrom dialout fax floppy lpadmin netdev plugdev sambashare syslog
XsessionErrors:
 (polkit-gnome-authentication-agent-1:17730): GLib-CRITICAL **: g_once_init_leave: assertion `initialization_value != 0' failed
 (polkit-gnome-authentication-agent-1:19225): GLib-CRITICAL **: g_once_init_leave: assertion `initialization_value != 0' failed
 (firefox:19333): GLib-WARNING **: g_set_prgname() called multiple times
 (firefox:19373): GLib-WARNING **: g_set_prgname() called multiple times
 (npviewer.bin:19431): Gtk-WARNING **: /usr/lib/gtk-2.0/2.10.0/engines/libqtcurve.so: wrong ELF class: ELFCLASS64
dmi.bios.date: 07/07/2008
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 080015
dmi.board.asset.tag: To Be Filled By O.E.M.
dmi.board.name: GF7050VT-M5
dmi.board.vendor: ECS
dmi.board.version: 1.0
dmi.chassis.asset.tag: To Be Filled By O.E.M.
dmi.chassis.type: 3
dmi.chassis.vendor: ECS
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr080015:bd07/07/2008:svnECS:pnGF7050VT-M5:pvr1.0:rvnECS:rnGF7050VT-M5:rvr1.0:cvnECS:ct3:cvr1.0:
dmi.product.name: GF7050VT-M5
dmi.product.version: 1.0
dmi.sys.vendor: ECS

[Angelos Vlassopoulos (avlass)]

Unfortunately I do not have a test environment, where I can test beta stuff.

However I made a new kind test and it turns out that if I boot with the elevator=noop and pci=routeirq options, there is a significant improvement.

I wrote a 2.1 Gbyte file with an average speed of 6.1MB/sec

[intropedro (intropedro2)]

I try a CD-live with the beta 2 of 10.04 and ubuntu hold at a good speed, but when you approach the GB began to fall about 6 Mb

[Ivan Terekhov (asm086)]

Very strange results when trying to copy a 656MB file on a FAT32 SDHC card through USB. And seems that it caused by the same issue with FAT32.

It starts copying pretty fast (at ~4MB/s), and after 30MB or so it starts to slow down, and down, and down...
After ~30 seconds it's ~1.5MB/s.
After first minute - ~1MB/s.
After ~30 minutes - ~120KB/s, and newer gets better.
For test purposes I divided the same file into 10 MB pieces, and got them all copied in ~2 minutes.
Exactly the same results on another, freshly-formatted to FAT32 Kingston 8GB SDHC (class 6), so I can't blame fragmentation.
The same Transcend card reader with the same SDHC cards works perfectly under WinXP.

AMD Athlon(tm) 64 Processor 3200+ / ASUS A8NE-FM / 1GB RAM
Ubuntu Karmic 32 bit with the latest updates.

[Patryk Szalanski (musefan)]

I am also experiencing very slow rates while transferring big files. When I copy a file to my USB storage device, it starts transferring with 30 MB/s and drops to 1 MB/s, which is really annoying, scince I don't want to boot to Windows 7 every time I want to copy something to my USB stick.

I don't know what information I should include into this post, so I am going to post the result of the following command:

uname -a
Linux p-ubuntu 2.6.32-21-generic #32-Ubuntu SMP Fri Apr 16 08:09:38 UTC 2010 x86_64 GNU/Linux

[rahman (darksky)]

I also experiencing this. Usually after reformating the USB stick the problem will gone.

[GilgongoJones (gilgongo)]

I'm on 10.10, uname -a: Linux monolake 2.6.35-22-generic #35-Ubuntu SMP Sat Oct 16 20:45:36 UTC 2010 x86_64 GNU/Linux, recently upgraded with a fresh install from 9.10.

When I plug in my Canon Ixus camera, I see a mixture of very, very slow transfer speeds, or sometimes very fast speeds followed by nothing at all, sometimes until the camera go to sleep and dismounts before the files can be transferred. With large files > 1Gb, I sometimes see no activity at all for about 5 mins, then it will suddenly start transferring very fast, then hang with only a few Mb to go, doing nothing.

[Sam_ (and-sam)]

Confirme with Maverick and now Natty. It starts file copy 350MB fast and slows down. USB-stick is FAT32. External USB-hd with ext4 and FAT32 is very fast.
Why it didn't bother until now, was due to rare usage of usb-stick. But recently reproducable CPU runs up high with plugged-in usb-stick ~83%, which isn't the case with usb-hd switched on. When opening palimpsest ~89% to unmount usb-stick I had a freeze. CPU fan goes from ~1400 to ~2200 rpm. Switched to tty, killed palimpsest and gvfs-gdu-volume, back to desktop CPU slowed down a bit, reopened palimpsest, CPU rises, unmounted the stick, closed palimpsest, CPU slows down to ~2-3 %.
Monitored with top and conky. No hints in logfiles.

[Brad Figg (brad-figg)]

This bug was filed against a series that is no longer supported and so is being marked as Won't Fix. If this issue still exists in a supported series, please file a new bug.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ing. Josef Klotzner (josef-klotzner)]

Hey Guys from ubuntu or linux!

You make it too easy.
It is no question of rasing a new issue or not.

It is 2013 now
you can use any version from 10.04 to 13.10 if you like
and

IT definitly C A N T never ever be, that windows is faster in file transfer as well with fat, fat32 or ntfs, but

it is a shame to all - it definitly STILL is ....
You might not see that the driver of fat32 or ntfs is a bottleneck if you are using an i7 running at 3 or more GHz.
BUT
as ALWAYS and everywhere in the world even with slowest and oldest CPU a USB 2.0 device is still slower and the REAL bottleneck a driver for a filesystem should NEVER EVER be slower than that.

But it definitly is.

Please escalate this bug if necessary directly to Linus Torvalds, because in my oppinion this must be immediatly improved.

IMMEDIATE PERFORMANCE FIX NEEDED FOR ntfs, fat and fat32 drivers

please forward to developers able to improve and fix

thank you

Ing. Josef Klotzner
Supervisor at well known austrian (worldwide growing) company