Slow swapin speeds after resume from disk

[Roger Binns]

I hibernate (suspend to disk). On resume the memory image is read back from the swap area at 70 megabytes per second. Then my applications start running and their memory comes back in from swap as well. For a large process like Firefox this takes a long time and Firefox is unresponsive until enough of its working set has been swapped back in. I can run "vmstat 1" and look at the "si" column to see how much data is being brought back from swap per second as well as "bi" to see the total number of blocks coming back in. For the vast majority of the time the numbers are identical meaning only swapped data is coming back in.

The swapped data comes back in at around 4 megabytes per second - a small fraction of the 70 megabytes per second from earlier. I usually resort to running "swapoff -a ; swapon -a" which runs at 10 megabytes per second and hence gets me working processes sooner.

Since I have 6GB of RAM, I would be very happy on resume from disk for swap to be pre-emptively put back into RAM at high speed. If the swapped data is being page faulted in (ie 4kb each time) then that is a really bad thing after a resume.

To repeat this, leave a terminal running "vmstat 1" and run Firefox opening a bazillion tabs and then hibernate. On resume try using Firefox while watching the terminal.

[Roger Binns (ubuntu-rogerbinns)]

Here is an example of "vmstat 1" output starting about 40 seconds after the resume (ie the desktop is mostly drawn but Firefox is still not responding):

  procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
   r b swpd free buff cache si so bi bo in cs us sy id wa
   0 1 1748664 5002420 7304 84452 4152 0 4152 12 269 477 2 1 50 48
   0 1 1748664 4998568 7304 84408 3848 0 3848 0 199 392 1 0 50 49
   0 1 1748664 4994588 7304 84444 3976 0 3976 0 245 454 1 0 49 49
   0 1 1748664 4990380 7304 84416 4144 0 4144 0 222 404 2 0 50 48
   0 1 1748664 4986144 7304 84444 4268 0 4268 0 249 445 1 0 50 48
   0 1 1748664 4982312 7308 84456 3860 0 3860 52 226 399 2 0 50 48
   0 1 1748664 4977972 7308 84452 4248 0 4248 0 248 463 1 0 50 48
   0 1 1748664 4974024 7308 84420 4016 0 4016 0 216 395 1 0 50 48
   0 1 1748664 4969900 7308 84400 4152 0 4152 0 252 443 1 1 50 47
   0 1 1748664 4965808 7308 84400 3988 0 3988 0 221 425 2 0 50 49
   0 1 1748664 4961592 7308 84444 4284 0 4284 0 262 485 1 0 50 48
   0 1 1748664 4957636 7316 84392 3980 0 3980 16 233 419 2 1 49 49
   0 1 1748664 4953564 7316 84404 4060 0 4060 0 256 452 1 1 52 45
   0 1 1748664 4947960 7316 84480 5572 0 5624 0 272 497 2 0 53 45
   0 1 1748664 4943756 7316 84480 4260 0 4260 0 261 454 2 1 47 51
   0 1 1748664 4940008 7316 84488 3728 0 3728 0 218 404 2 0 51 47
   0 1 1748664 4936412 7316 84444 3708 0 3708 0 260 434 2 1 53 44
   0 1 1748664 4932464 7316 84496 3932 0 3932 0 223 388 2 1 46 52
   0 1 1748664 4928640 7316 84492 3828 0 3828 0 249 455 1 0 54 44
   0 1 1748664 4924776 7316 84488 3876 0 3876 0 217 400 1 0 50 48
   0 1 1748664 4921036 7316 84472 3684 0 3684 0 262 462 3 2 47 48
   0 1 1748664 4917336 7316 84508 3672 0 3672 0 232 403 3 1 49 47
   0 1 1748664 4913720 7316 84464 3616 0 3620 0 260 440 3 1 50 46
   0 1 1748664 4910012 7316 84484 3728 0 3728 0 264 429 3 1 46 50
   1 1 1748664 4906280 7316 84464 3716 0 3716 0 271 468 2 1 49 48
   0 1 1748664 4903172 7316 84464 3076 0 3076 0 285 396 6 2 56 37
   0 1 1748664 4900072 7316 84508 3168 0 3168 0 339 468 8 1 51 40
   0 1 1748664 4880268 7316 84492 19804 0 19804 0 1295 2283 9 1 51 39
   0 1 1748664 4868472 7320 84620 11628 0 11760 0 732 1142 11 1 52 36

The last two seconds are when everything finally springs to life. Before that the machine is basically idle very slowly swapping things back in making me twiddle my thumbs.

[Dominik Stadler (dominik-stadler)]

I think this is basically the same thing as Bug 334536

[Dominik Stadler (dominik-stadler)]

To me the following items all report basically the same thing:

#329199 Slow swapin speeds after resume from disk
#334536 Hibernate flushes file caches (so afterwards my machine is slow)
#428554 Ubuntu is unresponsive directly after returning from hibernate
#455661 I/O slow after resume from suspend

Not sure if they can be made duplicates and which ones should stay open.

A possible "solution" would be to use TuxOnIce, which handles cache differently and thus does not have this slow repsonsiveness after resume.

See http://brainstorm.ubuntu.com/idea/1544/, everybody who would like to have that, should vote for that idea to indicate that this is an important feature that is missing.

[Roger Binns (ubuntu-rogerbinns)]

Note that there are important differences between these. For the file cache there is good reason to drop it. For example lets say you have an NTFS partition mounted, hibernate, boot into Windows (using the NTFS partition) and then reboot/resume back into Linux. If Linux continued to use previously cached information then it will trash the NTFS partition. And yes this really happened to me when I used suspend2 as TuxOnIce used to be known. Something similar would happen if you booted into a LiveCD and messed with existing hard disk contents (note that rescue mode likes to mount up everything you have). In this case I would much rather Ubuntu erred on the side of caution.

This ticket is specifically about the data of running programs. That cannot be discarded. The hibernation mechanism forces it all out from RAM into swap. Since the programs were using that data (which is why it was in RAM) they end up sucking it back in but using page faults which is extremely inefficient.

TuxOnIce saves file caches (dangerous) but also saves the data and compresses it. On resume it is all brought back into memory at once and so the programs can continue running, Any mechanism that puts back into RAM what was already there before hibernating would work well. Additionally the page faults cause random disk access which is very slow. If the data was brought back then the disk would be free for repopulating the file cache.

[Chow Loong Jin (hyperair)]

On Friday 30,October,2009 02:32 AM, Roger Binns wrote:
> Note that there are important differences between these. For the file
> cache there is good reason to drop it. For example lets say you have an
> NTFS partition mounted, hibernate, boot into Windows (using the NTFS
> partition) and then reboot/resume back into Linux. If Linux continued
> to use previously cached information then it will trash the NTFS
> partition. And yes this really happened to me when I used suspend2 as
> TuxOnIce used to be known. Something similar would happen if you booted
> into a LiveCD and messed with existing hard disk contents (note that
> rescue mode likes to mount up everything you have). In this case I
> would much rather Ubuntu erred on the side of caution.
I've learnt this the hard way before, but with swsusp (current method used in
Ubuntu), not TuxOnice/Suspend2, and using a vfat partition. I hibernated Windows
and booted into Ubuntu and vice versa. Next thing I knew, the paths were all
messed up (I had files with / in their names, and had to use photorec to do a
scan on the raw disk to retrieve its contents).

At the end of the day, what you should *never* do is mess around with a
filesystem that was not prior to hibernating. Or if you do, make sure that it
does not exist for the kernel (whichever you're using -- Linux, or Windows or
anything else) to play with when it resumes. This may not apply to filesystems
which have a journal, but then again I have no idea and have no wish to try my
luck again. Hence, I do not believe this has anything to do with the file cache.

--
Kind regards,
Chow Loong Jin (GPG: 0x8F02A411)
Ubuntu Contributing Developer

[Jason Smith (sddfdds)]

Although I use suspend instead of hibernate, I might be hitting this issue or something similar in natty. The reason I noticed is because I too have hard drive speed limited to 4MB/s.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[penalvch (penalvch)]

Roger Binns, thank you for taking the time to report this bug and helping to make Ubuntu better. The issue that you reported is one that should be reproducible with the live environment of the Desktop ISO of the development release - Vivid Vervet. It would help us greatly if you could test with it so we can work on getting it fixed in the next release of Ubuntu via http://cdimage.ubuntu.com/daily-live/current/ .

If reproducible, please execute the following via a terminal as it will gather necessary debugging information:
apport-collect 329199

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Reinaert Albrecht (reinaert-albrecht)]

In my experience this is solved by changing the swappiness value. Usually it is left at 60. It should be 10 or 15:
> cat /proc/sys/vm/swappiness

You can change it on the fly with
> sudo bash -c "echo -e 10 > /proc/sys/vm/swappiness"

and make it stick with
> sudo bash -c "echo 'vm.swappiness = 15' >> /etc/sysctl.conf"