Ubuntu slows down and hangs while copying file from/to USB

[Merlijn Sebrechts]

Hi

While copying many and large files to and from a USB drive, the system becomes incredibly slow and sometimes hangs. I watched the system monitor while it was running slowly but cpu was not above 50% at any time, the same for memory.

I'm using the "WD my passport" HD with a USB3 port of my "ASUS K55VD" laptop.

Description: Ubuntu 13.04
Release: 13.04

nautilus:
  Installed: 1:3.6.3-0ubuntu16
  Candidate: 1:3.6.3-0ubuntu16
  Version table:
 *** 1:3.6.3-0ubuntu16 0
        500 http://ubuntu-archive.mirror.nucleus.be/ raring/main amd64 Packages
        100 /var/lib/dpkg/status

Steps to reproduce:
1) Aquire many large files (like your totally legit moviecollection)
2) Copy files to external HD
3) Open your browser and be sad when your system becomes incredibly slow after a few minutes

This is my first bugreport so if I'm missing something, please tell me.

ProblemType: Bug
DistroRelease: Ubuntu 13.04
Package: nautilus 1:3.6.3-0ubuntu16
ProcVersionSignature: Ubuntu 3.8.0-27.40-generic 3.8.13.4
Uname: Linux 3.8.0-27-generic x86_64
ApportVersion: 2.9.2-0ubuntu8.1
Architecture: amd64
Date: Tue Aug 6 22:56:34 2013
EcryptfsInUse: Yes
ExecutablePath: /usr/bin/nautilus
GsettingsChanges:
 b'org.gnome.nautilus.window-state' b'geometry' b"'656x715+154+151'"
 b'org.gnome.nautilus.window-state' b'maximized' b'true'
InstallationDate: Installed on 2013-05-31 (67 days ago)
InstallationMedia: Ubuntu 13.04 "Raring Ringtail" - Release amd64 (20130424)
MarkForUpload: True
SourcePackage: nautilus
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Matiss Jekabsons (matiss-v)]

I am using Ubuntu 14.04LTS on Dell XPS 13 L321x and i am affected with this bug, too.

[Simon P. (simpre)]

I am affected by this problem too, running Ubuntu 14.04 64 bit.

"dirty files" seem to cause this problem on 64 bit systems. I solved it on my system by decreasing the dirty files cache. This article explains it in detail: http://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/

[zzarko (zzarko-gmail)]

As time goes by, my 14.04 64bit instalation performs worse and worse when doing anything with USB discs (flash or HD, doesn't matter).

If I start to copy a big file from/to flash USB, started programs usually work to a degree (if they don't require access to HD), but if I try to start something simple, like a terminal, during this, it will be started AFTER the copying was finished.

Similar thing happens when copying from/to USB HD, only not as severe as with Flash USB discs (the system is somewhat less laggy).

This is something that was frequent in my win xp days and after I switched to Linux in 2005 I tought that something like this is waaay behind me. Now, newer versions of Ubuntu are again showing me the joys of system lockup while working with USB discs.

[joseph tom (vamattom)]

@Simon P. (simpre) thank you...
64bit 14.04 is working fine after adding following lines

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

in /etc/sysctl.conf
and running sysctl -p

[Benji (sandoval-alvarez)]

Confirmed when I try to copy files to or frome my usb disck.

[Gachoud Philippe (ph-gachoud-gmail)]

@Simon P. (simpre) thank you...
Changed my life adding

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

in /etc/sysctl.conf
and running sysctl -p

************** lsb_relase *************
LSB Version: core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch:core-4.1-amd64:core-4.1-noarch:security-4.0-amd64:security-4.0-noarch:security-4.1-amd64:security-4.1-noarch
Distributor ID: Ubuntu
Description: Ubuntu 14.10
Release: 14.10
Codename: uto

[zzarko (zzarko-gmail)]

Tried that a while ago, it made working with already open applications and playing music OK (it stopped stuttering while copying), but if I copy something large and want, for example, to start a terminal window, I can click on terminal icon, but the window won't be shown until copying is finished. I mean, how much disk activity is needed for opening terminal window?

I have 3 harddisks in my machine. Let's say that OS is on HD1. The same problem with starting terminal occurs even if I start copying from HD2 to HD3, so I guess it has nothing to do with HD speed and/or caching.

And this is not only Nautilus problem (at least for me). Same thing happens if I copy/unpack from command line, or Double Commander (unpackig large archives does the same thing). Generally, every time some process has a lot of work with hard drive (it doesn't need to be an USB HD, that makes things just more sluggish), my computer is noticeably less usable until HD work is finished. I think that something is wrong with the way I/O operations are scheduled, as some processes can take all the machine's disk I/O resources for themselves, leaving absolutely nothing for others.

[Ubfan (ubfan1)]

The default cache setup on Ubuntu 14.04 (no explicit settings in sysctl.conf) seems to use bytes instead of a ratio:
$ cat /proc/vmstat | egrep "dirty|writeback"
nr_dirty 43
nr_writeback 0
nr_writeback_temp 0
nr_dirty_threshold 169462
nr_dirty_background_threshold 84731

This default is for a system with 8G memory, so are probably too small.

The problem caused by this default settings is slow file copies (10 MB/sec) from a SATA hd to a USB3 (SATA disk).

Using a ratio for the default settings instead of a number of bytes might allow better use of memory for machines with memory to spare.

[Manolis Kapernaros (kapcom01)]

Hello, today I experienced this problem for the first time. The only change on my laptop was the addition of 4GB RAM stick. Now I have 8GB total. I changed /etc/sysctl.conf as mentioned above ant the problem is solved.

[Nicolas_Raoul (nicolas-raoul)]

Reproduced every time on fresh install of Ubuntu 2015.10 on a Thinkpad T550 with 16GB of RAM, all settings by default except a few Unity tweaks, freshly rebooted.

When transferring a 364GB file, Ubuntu becomes extremely unresponsive. Mouse very difficult to move but I managed to click Firefox&s icon, it opened after a few minutes. On the opposite, I was unable to open a terminal despite clicking its icon several times, not sure why. The copy happens nonetheless, at 100 MB/second, and after it succeeds Ubuntu becomes usable again.

[Nicolas_Raoul (nicolas-raoul)]

Regarding my comment above: I realized that my problem happens also without any USB, just on the local disk, for instance when extracting a big tar file or deleting a directory that contains many files. Sorry if it is a different bug. Merlijn Sebrechts, do you observe the same problem on the local disk only?
Also, I have found a workaround: Whenever manipulating a big file, throttle disk usage by running commands in conjunction with `pv -L 5m`, the operation will take more time but at least the system will not hang.

[Fran (rsfmcj)]

Hi.

Same problem in my Dell Latitude D820, Centrino Duo, 32bits, 2GB RAM. Running Ubuntu Desktop 14.4 LTS.

After trying to copy 10GB data from my local SSD to a USB hard disk, the system freezes.

[Fran (rsfmcj)]

More feedback about the problem I'm experiencing.

First some data about my system:

Kernel 3.16.0-57-generic
Ubuntu Desktop 14.4 LTS
32 bits
2GB RAM
Centrino duo

I complete my previous comment. I was performing two I/O operations before I got the problem. I couldn't repeat the problem with the first one (that I described in comment #14), but It's always repeatable with the second one. That is:

1. Connect two USB disks to my computer to make a copy from the first one (say disk A) to the second (say disk B). The amount of data to copy is 80GB.
2. Start the copy from USB disk A to B. After 75% copying, the copy seems to be stopped.

The progress bar in "copy files window" doesn't move. You can close the window by pressing "x" with the mouse pointer. You can try to shutdown the system by pressing the "Shut down..." menu option, but it doesn't respond and system keeps online. Launcher bar doesn't repond.

I have by default the next values for dirty_background_ratio and dirty_ratio:

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

It seems to be that, during the huge copy from disk A to B, memory has reached the 10% of dirty pages indicated by vm.dirty_ratio. After that all I/O operations became synchronous and block the system until dirty pages a written to disk. It's just a supposition. Pending to chek it by increasing vm.dirty_ratio (so I allow the system to performance more asynchronous writes before they block the system) and repeating the copy.

[Christopher Barrington-Leigh (cpbl)]

When copying a large amount (multilple files) form an external drive (USB3 cable), my brand new Lenovo T450s laptop (20GB RAM, ... high specs) has frequent latencies/freezes. Doing nearly any operation takes several extra seconds, with zero responsiveness (e.g. trying to move mouse or switch to tty screen) in the mean time. Quite unusable. This is under the latest 15.10.

[snapy (sdfjsfjaei-hans)]

This also happens in 16.04 LTS when moving files. Mouse is very slow. Sometimes it doesn't move for many seconds.

[David Marques Neves (david-redeslinux)]

The problem is worst.

If you plug an USB external HDD and if have REALLY bad read problems (the drive in question is damaged and i'm trying to recover some data with ddrescue) in 120s, even if there is ANY disk, RAM or CPU load, the systems crashes EVERY TIME.

I try JUST TO PLUG this specific damaged drive in 6 servers and ALL just crashed after 2 minutes!

[Matthias Niess (mniess)]

This problem started ocurring for me when I moved to 16.04. I also upgraded my RAM from 8 to 16GB.

[Dmitriy (dimka-karp)]

The problem still exists in 16.04. And as I can see the ticket is not assigned to anyone...

[Saravanan (saran2somu)]

Same problem in my Laptop and PC also I have used Ubuntu 14.04 LTS version, both have 8 GB Ram, while coping 3GB file (a single Movie) to my mobile sd card or My 1 TB western external hard drive the system hangs and mouse pointer not moving . After restart the system only , system will normal.

[gladsonvm (gladsonvm)]

I am facing this bug. I tried to copy about 100GB to a toshiba portable hdd on ubuntu 16.04LTS live, and the whole laptop got frozen for more than 20 mins. But the lights that indicates disk activity on laptop and portable hdd is blinking. even right click was not working. I noticed this bug first when I tried to move about 100GB of data from /home/user/Documents and /home/user/Downloads to a portable wd passport drive. Whole thing was slowed down but for some seconds only and I didnt take it seriously.

Also I got a kernel update on my desktop yesterday. Does this new update solves his issue?

[Mauro (mauromol)]

I also experience this problem on Linux Mint 17.3 KDE (based on Ubuntu 14.04), copying through the KDE I/O subsystem (Dolphin or Krusader). So, I think Nautilus (which shows in the "Affects" field) is just a victim.
I have a notebook (HP ProBook 450 G1) with 4 GB RAM. I was thinking it might have been a problem with my laptop chipset (some sort of a bus saturation problem), but I see many have the same problem and that there's a workaround (I'll try as soon as possible).

Any official statement from Ubuntu developers?

[aliiv (aliiv)]

Also can confirm this bug on ubuntu gnome 16.04. didnt experience it on 15.10.

Copying big files to usb thumbdrive makes system hangs on unresponsive. very annoying

[Paul (paul1965)]

Same problem here, driving me crazy because it's unpredictable. Last night I copied 500GB (movies 1-2 GB) from an old external HDD to a new external external HDD. Finished the next morning without problems. Next I wanted to copy 5GB and the transfer became slower until it came to a halt. The system (Ubuntu 14.04) froze, I could't use the mouse/keyboard. Had to reset my computer at the risk of damaging the disk. Luckily that didn't happen. Tried a couple times but the system kept on freezing. Had to use Windows to get the job done.

[123vier (flowrist)]

This is still the case on 16.04.1.
System is a Laptop with an older Core i5, 4 GB RAM. While transferring some 200MB files the system is completely unusable.

At this point it's a little embarrassing not to be able to transfer files to a USB drive without getting system freezes in 2016 on a fresh install.
The workaround (vm.dirty etc) should not be necessary.

[Arno Mühren (arnomuhren)]

I can confirm this is the case on 16.04 too, using an external USB SSD.
This is on a Lenovo Thinkpad X220, Core i5, 8GB RAM.

[Quentin Engles (hollowdoor)]

This also happens for me, but it also eventually crashes my computer (AMD Athlon(tm) II X2 250 Processor × 2 with 4GB RAM).

[Michael MacEachern (maceach-b)]

I'm having this issue right now on 16.04. What's strange, is that if I copy say 600GB worth of thousands of files, about half way through, Xorg will start chewing up resources, and everything slows down to a hault, even though literally the only thing I ever opened was Nautilus. I never opened Firefox, or anything elses. Straight boot to the file manager to copy files. It's a weird issue to say the least, because I can't determine why Xorg would be freaking out on large file copying. If the files finish, the performance is still poor until I reboot the machine.

[Ivan Prisyazhniy (john-koepi)]

Having too on desktop ubuntu lts 16.04.1 with usb 3.0, new hdd, ntfs. Also, speed continuously slowing down and then freezes and power led of the disk start flapping.

[John Mapp (spidermonkey2012)]

I am running Mint 17.3 on an ASUS N550JX, am also having this issue. But the fix noted above did not resolve my problem.

_________________________________

vm.dirty_background_ratio = 5
vm.dirty_ratio = 10

in /etc/sysctl.conf
and running sysctl -p
_________________________________

Is there any other information regarding this issue? I see this issue when trying to transfer 10GB+ QEMU images.

[John Mapp (spidermonkey2012)]

Update: I applied the settings from the duplicate bug workaround, and that seems to have help a little. But still not 100%

vm.dirty_ratio = 60
vm.dirty_background_ratio = 40

[Julie Brandon (jewelie)]

Seeing the same problem since upgrading from 15.10 to 16.04.

The vm.dirty workarounds and disabling swap don't make a difference.

Once this problem kicks in ALL disk writes become stupidly slow (2MB/s) until next reboot on both SSD and spinning HDD.

i5-4460/16Gb RAM/ASUS Z97-K

This is an enormous problem not to have at least been assigned to someone!

[ullix (ullix)]

Same problem on Ubuntu Mate 16.04.01 with caja, nautilus and nemo.
Moving only a few dozen files can take several minutes. Files are on one disk in same directory tree.

Moving files with mv is almost instantaeous, too fast to measure.

Intel i74771 and SSD and HDs

[Michael Heuberger (michael.heuberger)]

Still having these slow USB performances after updating to Ubuntu 16.10 - and worse, it crashes sometimes when copying large files :(

[valk (vkhalil)]

Same here on ubuntu 16.04

[GTriderXC (gtriderxc)]

Slows down and hangs seem to be two different problems. My mashines never hang but "slow down" when copying thousands of different files. I found an interesting answer here:
http://askubuntu.com/questions/122113/copy-to-usb-memory-stick-really-slow

1'st answer from izx > reason 2, that You can see in my photo. After 3 hours of copying files I copied 29,4GB from 38,7GB but ONLY 20460 from 69502 files. If there is a problem that deserves a bug report then there is one with estimating the real time.
At first system showed me 4 hours, then decreased to 2, the last hour lasted 110 minutes. Copying itself (this screenshot was taken on 2016 but from Ubuntu 12.04) is very reliable I'd say if only USB drive is reliable.

[Dakota (fischmuetze)]

Same here on 16.04 and 14.04 ...

I made thousand of disk performance tests and found a lot of unspecific impacts of the performance without reasonable by any other process. Although the following tests are using a LVM vol this is also happen on a "normal" single device

$ hdparm -tT --direct /dev/mapper/VGHOME-LGHOME
/dev/mapper/VGHOME-LGHOME:
 Timing O_DIRECT cached reads: 2158 MB in 2.00 seconds = 1078.39 MB/sec
 Timing O_DIRECT disk reads: 2118 MB in 3.00 seconds = 705.58 MB/sec

$ hdparm -tT --direct /dev/mapper/VGHOME-LGHOME
/dev/mapper/VGHOME-LGHOME:
 Timing O_DIRECT cached reads: 294 MB in 2.03 seconds = 144.49 MB/sec
 Timing O_DIRECT disk reads: 178 MB in 3.03 seconds = 58.67 MB/sec

$ hdparm -tT --direct /dev/mapper/VGHOME-LGHOME
/dev/mapper/VGHOME-LGHOME:
 Timing O_DIRECT cached reads: 280 MB in 2.01 seconds = 139.10 MB/sec
 Timing O_DIRECT disk reads: 602 MB in 3.02 seconds = 199.42 MB/sec

For a long time we assume the virtual environment (VMware vSphere) but in the meanwhile I guess this is a ubuntu problem in special "hardware" environments - paired with larger volumes and a lot of inodes

[Alexander Morales Sevilla (ing-alexander-j)]

this bug affect me too, ubuntu 16.04.1

[akhilkalwakurthy (akhilkalwakurthy)]

i also have this bug in ubuntu 16.04.1, i have a dual boot system (i5-5200u 5th gen, 8gb). Copying files from windows directory was too slow, but it wasn't the case in 16.10.... (i had to revert because of some compatibility issues).

[Thomas (ew0)]

Regarding my previous comment - much better to add this line to a file in /etc/modprobe.d/

e.g. usbquirks.conf

Create this file and add

options usb-storage quirks=0x174c:0x1053:u

[Thomas (ew0)]

...and you get the HW address via lsusb (in my case 174c:1053)

[Blake Riley (cerran)]

I experienced this issue twice in the last 24 hours on 16.04. Both times, I waited over an hour after the computer froze before performing a hard reset on the machine. (The expected time for the operation to complete was under 10 minutes.) In my case, the computer goes directly from working as expected to being completely unresponsive, freezing all input (keyboard, mouse, power button) and output (screen, file transfer).

The bug does not appear consistently, though. After restarting the computer, performing the same file transfer worked without any issue.

[kutuzof (kutuzof)]

@ew0 I setup the usbquirks.conf the way you recommended. I also confirmed that the address is right with lsusb. However it doesn't seem to have had any effect. How can I confirm that it's being loaded?

[Rudy Fernandes (rudolf-fernandes)]

I'm not sure whether the following is clear to whoever is looking into this bug: The issue arises only when using a USB3 port; it does not happen when using USB2. At least, that's the way my 14.04/12 Gb system is affected.

[Vishal Kute (vishal.kute)]

I've tried to copy my data (2GB)to the external drive, it takes almost 4-5Hours to finish. I'm on Ubuntu 16.04 LTS 4.4.0-130-generic

[mrvst (maravento)]

Same problem in Ubuntu 18.04.1 x64 (Gnome and Mate Confirmed). None of the proposals described work

[John Bell (jkabell)]

I have run into it with Ubuntu 18.04.1 x64 also. I have a completely separate drive and Windows system in the same computer and Windows, dare I say it, has no problem and is quick. My data files that fail are 400-500MB and they freeze after about 200MB of transfer when copy from a 32GB SanDisk Ultra (80MB/s) SD card. But success with Windows tells me I don't have a hardware problem.

This bug is now more than 3 years old. Like #78, I am disappointed.

[Sebastien Bacher (seb128)]

Is the issue there specific to nautilus or do you have the same if you copy from a command line (using cp or gio cp)

[Facundo Batista (facundo)]

No matter how you do the copy, the problem presents.

[Sebastien Bacher (seb128)]

So even if you use the 'cp' command line tool?

[Facundo Batista (facundo)]

Yes. It's how I copy stuff 99% of the times, anyway.

`cp ~/movies/BigSomething.mkv /media/facundo/PenDrive/`, and the whole system starts to lag.

[Sebastien Bacher (seb128)]

Thanks, it's not an issue in nautilus then, reassigning to linux

[Remona Minett (rminett)]

Just began dealing with this issue in Ubuntu 19.04. Basic fresh installation. Copying ~1 TB between Hard Disk A and Hard Disk B caused the entire system to stop responding for minutes at a time. While it was 'responding' the mouse movement on-screen stuttered badly and drifted. Keyboard input is completely ignored. The data transfer is between two SATA drives. Copying from a USB stick has produced the same effect, however the data was ~120 GB. Using both cp as well as the file manager produces this effect. I notice the delay in response time in the terminal during these copy jobs, when attempting to type anything, the delay in a response in the system is nearly a full minute. I WAS able to reproduce this on my laptop as well, running the same fresh installation but with 24 GB ram and 16 CPU threads, from SSD to SSD instead of HDD to HDD.

AMD FX-6300 (6 cores)
8 GB DDR3 1600 ram
GeForce GTX 1060 6GB
64-bit Ubuntu 19.04

[Michael Z Freeman (michael-z-freeman)]

Hi. This needs to be fixed. This caused me no end of problems over the years. It makes Ubuntu look bad. I thought it was a problem with my hardware until I found - https://blog.programster.org/fix-freezes-when-transferring-files ...

"It appears that this has to do with having a very large cache of "dirty files" being held, and when that cache gets too full, your system will "pause" whilst it ensures all that data is actually written to disk. Thus to "fix" this issue, you can reduce your cache size so your system doesn't get overwhelmed to the point where it becomes unusable."

The defaults need changing.

[Geoffrey (trentortrentor)]

I recently replaced Win 8.1 with Mate Ubuntu 18.04.3 LTS. Copying large files with Unison or Freefilesync (e.g., 1.4gb wmv) causes freeze. This happens when copying via internal HD to External usb 2.0 HD as well as internal to internal. When the freeze happens, all devices connected on the same home LAN get disconnected from internet and asus router admin page shows an X indicating disconnected from internet. If I power off the frozen pc, all LAN devices immediately reconnect. I have tested it with minimial apps running (e.g., virtualbox off, syncthing off), and the problem still occurs. I have never had the LAN issue before. I have not yet had the opportunity to try the fixes in the above discussion because I can't afford to get disconnected from the home equipment while at the office. Crossing fingers that one of the fixes will work. Also, I'm wondering if the issue also exists in Ubuntu server.

[Tronde (tronde)]

Wow, six years and still now solution or even someone working on this. I suffer from this bug on the latest versions of 16.04 and 18.04. Newer versions I did not test, yet.

What is needed, that this bug is processed, triaged and maybe resolved?

[Mike (0x656b694d)]

Ubuntu 19.10 here. UI, including mouse, freezes on copying big files within the same SSD, but not with HDD.
The suggested change of

  vm.dirty_background_ratio = 5
  vm.dirty_ratio = 10

doesn't help.

Samsung SSD 850 EVO 250GB (EMT02B6Q), SATA, not hotplug.

SMART overall-health self-assessment test result: PASSED

ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)

The drive has two ext4 partitions: for the system root and for media files.

[gunessee (gunessee)]

Ubuntu 18.04 here.

Copying big files from SSD to usb device takes too long after the copy percentage reaches 99%.

[peterrus (petorrus)]

I have been running some comparative tests between distro's on a Thinkpad T450S with a i5-5300U and I think this is related to CPU throttling.

The issue seems to occur on:
 - Ubuntu 18.04 (5.0.0 stock kernel)
 - Ubuntu 19.10 (5.3 stock kernel)
 - OpenSUSE Tumbleweed (5.3 stock kernel)

But not on OpenSUSE 15.1 Leap (4.12.14 stock kernel)

I have installed the mainline 4.12.14 kernel on Ubuntu 18.04 but that gives me the same issues.

What I have noticed however is that 18.04, 19.10 and Tumbleweed all try to throttle the CPU, and during file copy the CPU frequency never gets above 1Ghz.

OpenSUSE Leap however has a much more lax throttling policy and never dips below 2Ghz, even when the laptop is running on battery.

I have checked the CPU frequency with `cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq`. Also this tool might give some valuable insight: https://github.com/amanusk/s-tui

I find that running a stresstest with that tool (or stress-ng, which with it integrates) seem to eradicate the issue (because the CPU frequency is kept unthrottled while running a stresstest).

If anyone has some ideas for tests I can do, please let me know. I have a spare SSD that I can pop in the laptop so it shouldn't be that big of a deal reinstalling the OS.

[peterrus (petorrus)]

Alright, I have run some more tests:

I copy a file in my (unencrypted, ext4, on root partition) homedir with:

dd if=testfile_8gb of=testfile_out status=progress

(I made the testfile using dd if=/dev/zero of=testfile_8gb bs=1M count=8000 status=progress)

While running this Xorg/Gnome3/Mouse/Any UI application becomes unresponsive for seconds at a time, with intervals of a few seconds where everything is responsive. The write speed of dd often starts somewhere between 200 and 100 mb/s and then dips all the way to around 35, after which the dd process completely locks up. (I think this is because the process gets stuck into some kind of iowait state, I deduced this from some quick peeks at KSysGuard, citation needed). After a few minutes the dd process suddenly finishes and the UI becomes responsive again. Copying the file through nautilus has similar behaviour.

Disabling intel_pstate via the kernel boot parameters (intel_pstate=disabled) did not have any effect (except for indeed disabling intel_pstate).

Doing some tweaks to the vm dirty bytes settings had no significant effect (making the mouselag more of a jitter, but still unusable).

Disabling the swapfile (19.10 has a swapfile instead of swap partition by default) however completely mitigated the issue!

If some people are willing to try to reproduce this we might be able to get something going:

- Open some Youtube video in Firefox (something that lets you see if the UI lags)
- next to that open two terminals, in one terminal start the dd process as described above
- The ui should be lagging now, mouse doesn't move for seconds at a time, youtube video severaly lags/hangs/jitters
- in the second unused terminal run 'sudo swapoff /swapfile'
- For me the lag immediately stops
- Killing the dd process/letting it finish and then starting a new one works perfectly, much higher writespeeds (and more stable)
- enable swag again with 'sudo swapon /swapfile'
- Lag returns completely

I am really curious if this works for others. Maybe this is an entirely different issue, for which I will file a bugreport if nobody can reproduce this.

Thanks in advance!

[peterrus (petorrus)]

I have tried to eliminate using a swapfile (as is default since Ubuntu 18.10) instead of a swap partition. Results are as follows:

Using a swap partition (10GB) instead of a swapfile seems to increase dd's writespeeds with about 10% and seems to decrease the amount of stutter and lag, but doesn't completely stop it from happening. Again, doing a swapoff seems to completely eliminate the issue.

[peterrus (petorrus)]

After upgrading my RAM from 12GB to 20GB I get significantly better results. Even with swap enabled I get a lot less UI lockups, and even when I get them, there are significantly shorter.

This might be logical as I the issue seems to happen when swapping and because of the increased amount of RAM less swapping occurs. But I am just guessing here. I hope this can help someone somehow.

[Lukas (luckyluke751)]

i can confirm the behaviour described by peterrus (#107)

[Merlijn Sebrechts (merlijn-sebrechts)]

Thanks @peterrus!

I can confirm that using swapoff completely removes the hanging and stuttering during heavy disk IO.

I don't see this only during file copy. Extracting archives, for example, or saving virtual machine state to disk cause the same kind of stuttering and hanging. It was impossible for me to import an .ova image into VirtualBox when swap was on because my system became unresponsive. After turning swap off, I can use virtualbox again!!

[gfx (oce)]

On my desktop i5-4690 with 8gb ram copying from usb3 to disk is still slow and blocks playing video on YouTube. Happens with dolphin and cp in the terminal. Running kubuntu 20.04

[gfx (oce)]

A sudo swapoff -a mid copy solved the response issue, video is playing again...

[Merlijn Sebrechts (merlijn-sebrechts)]

I feel like this is a lot better on Ubuntu 20.04. Anyone else have the same experience?

[Mike (0x656b694d)]

Me too. I turned the cache back on and don't notice the freezes.

[rodan (smrodin)]

My day was saved by #107 comment, I tried everything mentioned in this issue, and always get a lagged mouse!, mi machine es a laptop lenovo L480 12GB RAM, with SSD ( a very capable machine), after I deactivate my swap partition (20GB) (swapoff -a) my system starts working as a champ, thanks @peterrus (petorrus)!

[Koopee (koopee1234)]

I don't know if this helps, but I have experienced similar issue. Might be a different reason, though.

1. connect phone (Oneplus 6) to computer and start large file transfer from there (~10GB or so).
2. Insert memory card to memory card reader on the computer
3. The system is really unresponsive until the phone transfer completes.

Ubuntu 18.04 with 5.4.0-42 kernel

Linux 5.4.0-42-generic #46~18.04.1-Ubuntu SMP Fri Jul 10 07:21:24 UTC 2020

[Yug (hugolpz)]

(This post uses md syntax),
### Software
Ubuntu 20.04. <---- bug still on 20.04. Agree with #103 #101 #94 #78

### Hardware
Several posts above mentioned hardware, RAM or RAM updates, so I will document mine as well.

Initial > ASUS K401UB :
- Intel Core i5-6200U (2.3GHz)
- 4GB DDR3L
- 1TB HDD,
- NVIDIA GeForce 940M 2GB DDR3

Improvements > RAM :
- Crucial CT102464BF160B : 8Go (DDR3L, 1600 MT/s, PC3L-12800, SODIMM, 204-Pin)
- Product page: https://www.amazon.com/gp/product/B006YG8X9Y
- Date: 2 months ago.
- Now: 4+8=12GB.

### Symptomns:
Past week, I copy large GIS files to my SSD. It starts super fast as expected, then fan runs wild and either :
- transfer and UI slows down
- transfer freeze temporarily, then restarts ; UI is fine
- transfer and UI freeze temporarily, then restarts
- transfers and UI totally freeze, after 10mins+ I reboot

This `top` output was when it was lagging:

$top
top - 12:11:32 up 1 day, 18:23, 1 user, load average: 6,77, 4,46, 3,81
Tasks: 320 total, 5 running, 315 sleeping, 0 stopped, 0 zombie
%Cpu(s): 66,0 us, 13,7 sy, 0,0 ni, 6,9 id, 13,4 wa, 0,0 hi, 0,0 si, 0,0 st
MiB Mem : 11880,7 total, 156,0 free, 5664,9 used, 6059,8 buff/cache
MiB Swap: 0,0 total, 0,0 free, 0,0 used. 5324,5 avail Mem

### My little secret dirty hack
While copying, unmount the disk. The copying UI box pop an error.
Under the hood it seems to clears something.
Mount back the disk. Then on the copying UI box pop an error, click "retry".
It start again at full speed.

### Observations
This shows again that it's not a single file bug but a cache/cumulative issue.

While solid leads are shown above, no solution above solved it all so far.
After 6 years it may needs a professional dedicated on the case
for a definitive diagnostic and fix.

[Yug (hugolpz)]

[EDIT]:
* PC hard drive is an SSD as well (upgraded 2 years ago).
* Transfer speed from internal SSD to external SSD. (See attached image)
** declared: 40MB/s
** observed: 12kB/s (I hand measured 87sec/1MBs)
** slowing factor: 3000+
* Dirty little hack : unmount then remount external disk hack did NOT work when done over heavy 457MB GIS file. Likely over flooded the cache as it started.

[Yug (hugolpz)]

# Objective
Monitor dirty files memory,
Change to more suitable values.

What it does: change from default 10 and 50%.

My understanding:
If `(vm.dirty_background_ratio/100) * memory size > largest file size`, then it should be ok.[reference needed]

Note: `memory size` may be *available* memory size. Closing your memory-eating browsers may help.[reference needed]

After editing the dirty values, reboot your system. It helped me.

# Sources:
- https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/
- http://fooo.fr/~vjeux/github/github-recommandation/db/doc/manual/html/linux-performance-guide.html

# Helpful commands
```
# View one dirty setting
sudo sysctl -n vm.dirty_ratio # default 10
sudo sysctl -n vm.dirty_background_ratio # default 50

# View/monitor several settings
cat /etc/sysctl.conf | egrep "dirty|writeback" # config file's dirty, if loaded.
cat /proc/vmstat | egrep "dirty|writeback" # print dirty
cat /proc/vmstat | egrep "nr_" # print all nr

# View/monitor several settings : dynamic
watch grep -A 1 dirty /proc/vmstat

# Soft edit dirty ratios
sudo sysctl vm.dirty_background_ratio=50 # set variable's value
sudo sysctl vm.dirty_ratio=80 # set variable's value

# Hard edit config file
sudoedit /etc/sysctl.conf # hard edit config file.
#vm.dirty_background_ratio = 50 # add without the hashtag
#vm.dirty_ratio = 80 # add without the hashtag
# # Save with nano : CTRL+X, then y
sudo sysctl -p # reload current config file. Resets soft variables.

# Check changes
cat /etc/sysctl.conf | egrep "dirty|writeback" #
cat /proc/vmstat | egrep "dirty|writeback" #
```

# Proposed diagnosize for your steps
Before and after changing dirty values.

```
# 1. System is waiting
cat /proc/vmstat | egrep "dirty|writeback"

# 2. Just started copy
cat /proc/vmstat | egrep "dirty|writeback"

# 3. Copy is staled
cat /proc/vmstat | egrep "dirty|writeback"

# 4. Just cancelled copy via mouse
cat /proc/vmstat | egrep "dirty|writeback"

# 5. Waiting = nr_dirt goes down
cat /proc/vmstat | egrep "dirty|writeback"

# 6. Back to normal.
cat /etc/sysctl.conf | egrep "dirty|writeback"
```

[Yug (hugolpz)]

[EDIT]
I still have the but if my Chrome/Chromium and associated 30 tabs have been started. Even if I close them. Even has htop shows I just have 2.5G/11.6G used.

Duplicate of https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1267648 which provides better information on how to replicate on any machine.

Interestingly, it says: "I discovered two more requirements to reproduce this bug: free RAM should be low and swap must be enabled."

[unnameless (unnameless)]

Can't believe this bug dates all the way back from 2014 and 8 years later in 2022 this is still a thing! Not only that the bug has nobody assigned to it and is of "undecided" importance!

My experience is appalling even by juggling with the vm.dirty_ratio and other dirty bytes values and my system is nothing to sneeze at (2 x 7763 Epycs, 2TB RAM etc). Trying to copy 16TB of data, that consists of big files (tens and even hundreds of GB of files), to a USB 3.0 RAID0 drive, would start initially at ~300MB/s copy speeds (cp, rsync...doesn't matter) and fall quickly to like 10% of that value of 30MB/s as dirty bytes values soar as soon as copy begins. Bumping the drity ratio to even 99, meaning filling up the RAM with the dirty values, doesn't help much either as it only block the whole RAM but also allows for only a few files to be copied at full speed, the throttle still being applied as soon as the threshold is reached. To get back at full copy speeds, one needs to either wait for dirty values to go down (ages) or find a way to clear the RAM cache (hard reboot would do it but it's unacceptable).

This BUG is simply incredible and prof that Linux and the kernel more precisely has yet to mature itself even after 30+ years

[LAZA (laza74)]

I can report, that i have the problem now on my NAS system.

What a mess...

[zzarko (zzarko-gmail)]

I haven't had a problem with this for a long time, but it started again about a month or two ago, on 22.04 system.

It does not matter what is the source and/or destination (NFS shared drive, local HDD, local SDD, USB, whatever), or is same or different drive used as source and destination, when I start a copying operation the whole system just hangs, mouse can barely move, music stops to play or is very choppy. System is back to be usable only after copying has finished.

I applied /proc/sys/vm/dirty_background_bytes and proc/sys/vm/dirty_bytes trick, it helped a bit, but did not solve the problem.

[Ali-Hashemi (io-geek)]

ubuntu 22.04 and 1TB hdd with a considerably fine hardware status.
problem still remains, matter of fact i just lost a whole bunch of data because after screen froze i just unplugged the usb and then i had to re-create the ntfs volume from scratch (gparted) to be able to mount it again.