KDE GUI freeze on high disk IO

[JPT]

I've got massive problems with GUI on my PC.

The attached logs are from the following setting:
- high disk load (f3write)
- kernel 4.15.0-1050-oem (the only 4x kernel available in eoan)
- no swap available

These are the parameters (disk load, kernel, swap) I am fiddling with to find out the cause, see below.

In general
- the problem happens once a day with any 5.0 or 5.3 kernel from eoan repo
- more likely to happen with high disk or CPU load, but not necessarily.
- less likely with the 4.x kernel mentioned above.

I just installed the following kernels from kernel-ppa:
5.3.10-050310-generic
5.4.0-050400rc6-generic
5.4.0-997-generic (drm-intel-next)
Now I am waiting for the freeze to occur again.

The problem started about half a year ago.
There have been three changes that happened at the time and could be the cause
- switch to kernel 5.x
- update to Ubuntu 19.4
- swapped motherboard, CPU and RAM.

The hassle started with the following scenario, repeated often:
- high disk load
- PC becomes laggy
- soon dies/freezes

The longer I waited, the deeper the lockdown went. but I don't know the exact details.
The first cause I found was kswapd running amok and take the system with it. Workaround: add swap space.

Now the problem shifted: Whenever I got high disk load (eg dd 100 GB of data) the kernel swaps likle crazy and makes the GUI first lag soon freeze. but no excessive CPU load any more, as kswapd does not go crazy.

Again I found a workaround:
the above mentioned 4.x kernel. In this case the GUI lags as expected from a heavily swapping system. nothing more.
but still sometimes the GUI simply freezes (the error reported at the top)

Another workaround is to completely remove swap. Let's see what happens, if kswapd goes amok again.

I am not sure as how these problems are related.

I did extensive hardware check like
- memtest86 newest version, no "may be vulnerable to high frequency row hammering"
- smart

do you suggest any burn in test suites?

thanks

ProblemType: Bug
DistroRelease: Ubuntu 19.10
Package: xorg 1:7.7+19ubuntu12
ProcVersionSignature: Ubuntu 4.15.0-1050.57-oem 4.15.18
Uname: Linux 4.15.0-1050-oem x86_64
ApportVersion: 2.20.11-0ubuntu8.2
Architecture: amd64
Date: Sun Nov 10 15:55:14 2019
DistUpgraded: 2019-10-19 13:35:30,394 DEBUG entry '# deb http://ppa.launchpad.net/brandonsnider/cdrtools/ubuntu cosmic main # disabled on upgrade to eoan' was disabled (unknown mirror)
DistroCodename: eoan
DistroVariant: ubuntu
DkmsStatus:
 virtualbox, 6.0.14, 5.3.0-18-generic, x86_64: installed
 virtualbox, 6.0.14, 5.3.0-19-generic, x86_64: installed
ExtraDebuggingInterest: Yes
GraphicsCard:
 Intel Corporation UHD Graphics 630 (Desktop 9 Series) [8086:3e98] (prog-if 00 [VGA controller])
   Subsystem: Micro-Star International Co., Ltd. [MSI] UHD Graphics 630 (Desktop 9 Series) [1462:7b16]
InstallationDate: Installed on 2017-12-16 (693 days ago)
InstallationMedia: Kubuntu 17.10 "Artful Aardvark" - Release amd64 (20171017.1)
MachineType: Micro-Star International Co., Ltd. MS-7B16
ProcEnviron:
 LANGUAGE=en_US:en
 TERM=linux
 PATH=(custom, no user)
 LANG=de_DE.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-1050-oem root=UUID=ca93bfbd-335a-4cc7-9d0f-f0da18447497 ro quiet splash vt.handoff=7
SourcePackage: xorg
Symptom: display
Title: Xorg freeze
UpgradeStatus: Upgraded to eoan on 2019-10-19 (22 days ago)
dmi.bios.date: 08/13/2019
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 2.80
dmi.board.asset.tag: Default string
dmi.board.name: B360 GAMING PRO CARBON (MS-7B16)
dmi.board.vendor: Micro-Star International Co., Ltd.
dmi.board.version: 1.0
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Micro-Star International Co., Ltd.
dmi.chassis.version: 1.0
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr2.80:bd08/13/2019:svnMicro-StarInternationalCo.,Ltd.:pnMS-7B16:pvr1.0:rvnMicro-StarInternationalCo.,Ltd.:rnB360GAMINGPROCARBON(MS-7B16):rvr1.0:cvnMicro-StarInternationalCo.,Ltd.:ct3:cvr1.0:
dmi.product.family: Default string
dmi.product.name: MS-7B16
dmi.product.version: 1.0
dmi.sys.vendor: Micro-Star International Co., Ltd.
version.compiz: compiz N/A
version.libdrm2: libdrm2 2.4.99-1ubuntu1
version.libgl1-mesa-dri: libgl1-mesa-dri 19.2.1-1ubuntu1
version.libgl1-mesa-glx: libgl1-mesa-glx 19.2.1-1ubuntu1
version.xserver-xorg-core: xserver-xorg-core 2:1.20.5+git20191008-0ubuntu1
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev N/A
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:19.0.1-1ubuntu1
version.xserver-xorg-video-intel: xserver-xorg-video-intel N/A
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:1.0.16-1

[JPT (j-p-t)]

I exactly once got a GPU HANG. Wasn't able to fetch the logs though and did not get one again:

14:11:58 i915 0000:00:02.0: GPU HANG: ecode 9:0:0x00000000, hang on rcs0
14:11:58 GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
14:11:58 Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
14:11:58 drm/i915 developers can then reassign to the right component if it's not a kernel issue.
14:11:58 The gpu crash dump is required to analyze gpu hangs, so please always attach it.
14:11:58 GPU crash dump saved to /sys/class/drm/card0/error
14:11:58 i915 0000:00:02.0: Resetting rcs0 for hang on rcs0

and this problem happens regardless if xserver-xorg-video-intel is installed or not.
but maybe the GPU hang log occurs only if the Intel driver is installed?

[JPT (j-p-t)]

Does not seem to occur in this setting:
- kernel 5.4.0-997-generic
- swap available
- high disk and cpu load
- xserver-xorg-video-intel installed

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[JPT (j-p-t)]

Occurs after a few seconds with this setting:
- Kernel 5.3.10-050310-generic
- swap available
- moving a 10 GB partition with gparted (high disk load on a USB attached SSD)
- xserver-xorg-video-intel installed

disabling swap solved the problem.

also the mouse hangs when window animations are active.

[JPT (j-p-t)]

mouse lagggggssss. almost unusable

- kernel 5.4.0-050400rc6-generic
- medium load network traffic (20MB/s) and accompaning disk traffic
- swap available
- xserver-xorg-video-intel installed

again, disabling swap solved the problem.

this sucks because I hoped with the release of an official ubuntu 5.4 kernel the problem might disappear.

[Daniel van Vugt (vanvugt)]

Which GUI are you using that exhibits the problem?

[Daniel van Vugt (vanvugt)]

Also please run this when the problem is happening:

  ps auxw > processes.txt

and attach the file to this bug.

[JPT (j-p-t)]

Which gui? well: KDE, Xorg, Intel driver, sddm. What exactly are you asking for?

I had a freeze again and had the syslog running in a toplevel window with GRUB_CMDLINE_LINUX="drm.debug=0x1e log_buf_len=1M
Did not get any message that would explain the freeze. And did not ever get a GPU HANG again.

Will fetch the list of running process on next freeze.

[JPT (j-p-t)]

I am not sure if there are two different problems:

- the lagginess (repeated freezing for let's say some 100ms) when increasing cache/buf makes the kernel swap too much (which is clearly different from a system that suffers from heavy swapping, far worse)
- the complete and final freeze of the GUI. this happens regardless of swap and probably regardless of disk or cpu load.

[JPT (j-p-t)]

attached "ps auxw" while GUI was freezed.
kernel 5.4.1-050401-generic
no swap
I restarted GUI using /etc/init.d/sddm restart
right now only the KDE taskbar and menu are locked, Alt-Tab and programs still work. (this is a rather new phenomena)
I already experienced this once, where some programs were freezed too (eg libreoffice) and others weren't. I only got a message like "program does not answer, force close it"?
maybe this is a second hand problem as some inter-process communication locks apps when KDE is locked?

after about 5 minutes KDE bar crashed and restarted. now works.

from syslog while KDE bar locked:
pulseaudio[30539]: E: [pulseaudio] bluez5-util.c: GetManagedObjects() failed: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
...
PackageKit: daemon quit
systemd[1]: packagekit.service: Main process exited, code=killed, status=15/TERM
systemd[1]: packagekit.service: Succeeded.
...
dbus-daemon[1772]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.679' (uid=1000 pid=32050 comm="/usr/bin/plasmashell ")
systemd[1]: Starting PackageKit Daemon...
PackageKit: daemon start
dbus-daemon[1772]: [system] Successfully activated service 'org.freedesktop.PackageKit'
systemd[1]: Started PackageKit Daemon.

[Daniel van Vugt (vanvugt)]

Thanks. This looks like the cause of high disk IO at least:

jan 3400 0.0 0.0 2600 676 pts/8 S Dez07 0:00 /bin/sh /usr/bin/bcompare RAID/Incoming/NAS-DATEN/ jan/NAS-DATEN/
jan 3412 70.7 11.3 2759972 1851804 pts/8 Sl Dez07 900:13 /usr/bin/bcompare RAID/Incoming/NAS-DATEN/ jan/NAS-DATEN/

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Kai-Heng Feng (kaihengfeng)]

Please test latest mainline kernel:
https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.5-rc1/

[Daniel van Vugt (vanvugt)]

I'm not really sure this is a bug. It sounds like expected behaviour:

1. Disk gets thrashed.
2. Any processes relying on memory swapping to/from that disk get delayed.

No particular process is doing anything wrong. Just if you want to avoid that situation then maybe don't configure swap on such a disk.

Then if you can survive without swap, problem solved. If you still need swapping for other reasons then either do so on a different disk or add more RAM, or run fewer programs.

I don't think there's any bug here(?)

[Kai-Heng Feng (kaihengfeng)]

According to the description there's no swap in the end, so I think it's like discussion in [1]. So yes, I think it's still a kernel bug.

For desktop usage, [2] [3] are mentioned by dsd in the discussion. Maybe we can adopt those daemon as an OOM killer for desktops.

[1] https://lkml.org/lkml/2019/8/4/15
[2] https://github.com/endlessm/eos-boot-helper/blob/master/psi-monitor/psi-monitor.c
[3] https://gitlab.freedesktop.org/hadess/low-memory-monitor

[JPT (j-p-t)]

There is for sure a bug.
As I said, I know how a system behaves that is suffering from too much swapping.

My case is far worse.

and still, about once a day the GUI is completely locked. this cannot be explained from swapping and cannot be solved by disabling swap.

I think the only kernel yet that did not suffer was

[JPT (j-p-t)]

sorry...
I think the only kernel yet that did not suffer was
https://kernel.ubuntu.com/~kernel-ppa/mainline/drm-intel-next/
5.4.0-997_5.4.0-997

will try 5.5-rc1

[1] is not totally equal, but may be related.

[Daniel van Vugt (vanvugt)]

Incomplete, pending investigation per comment #18.

[JPT (j-p-t)]

Haven't had a problem for over a week now with kernel 5.4.1-050401-generic.
Strange, as a comment above says it happened with this kernel as well.

So it probably was fixed by the Firefox update?

updates since Dec 8th contain:
- Firefox from 69.0.3 to 71.0
- Thunderbird
- samba, git, wine, ssh, makemkv

[Daniel van Vugt (vanvugt)]

I think more likely it was a kernel bug and we just haven't bisected it clearly yet. Try going back to a v5.3 kernel like Ubuntu is shipping and see if it happens there?

[JPT (j-p-t)]

Still this bugs haunts me every day.

kernel 5.3.0-29-generic still affected.

whenever swap is enabled, the system is almost unusable when cache grows too large with time. as already told, this is by far worse than the infamous linux swap trashing.

while this can be worked around easily, I am still logged out without any warning randomly about every day, killing all active programs.
This seems a bit more likely when I have Windows Virtualboxes running, that have more than one processor core assigned. I already updated VB-Client, VB-Settings...

my assumption: this is some threading issue. maybe wrong priority of some graphics stuff, maybe too many semaphores in swap-in code... I don't know.

This problem annoys me for almost a year now. I give up.
My last try is making Virtualbox work with the testing kernels.
If this doesn't work I will try some other distribution.

[Daniel (danito8905)]

Hi, I had the same problem, on high disk IO (SSD) including swap.

I have install this version of kernel and the problems is gone for the moment.

https://kernel.ubuntu.com/~kernel-ppa/mainline/drm-intel-next/2020-02-26/

[Daniel van Vugt (vanvugt)]

I have a feeling based on experience with multiple machines running gnome-shell that this issue was specific to kernel 5.3 and has gone away in 5.4 and later.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[JPT (j-p-t)]

Probably fucked up installation was the cause of this. sorry for bothering.

I recently reinstalled Kubuntu because I was frustrated with the stability of my system. It still hadn't really improved since switch to kernel 5.4
And, magically, all the crashes are gone. So, no idea why but something in my old system seems to have caused most of the crashes.

[Chris Cheney (ccheney)]

I have this problem as well on Cinnamon, so it doesn't appear to be specific to KDE. It stalls often and frequently over 15-30 seconds at a time for no apparent reason. But the swapping to disk may be a clue.

I see this on 5.4.0-40 and have also tried with the lowlatency version of the kernel and it doesn't help.

One thing that seems to reduce the frequency of the problem somewhat is to disable transparent_hugepage support but it still happens even after that.

I have 32GB ram and it starts happening when I get to about 50% ram (mem available) usage. The system also appears to start swapping at that point despite what swappiness is set to. I tried setting it to 10 but it is still swapping at what appears to be ~ 50% usage.

It appears to be very easy to reproduce if you run a few VMware VMs on a system.

# cat /proc/sys/vm/swappiness
10

$ free -m
              total used free shared buff/cache available
Mem: 31030 6591 246 9050 24192 14936
Swap: 32767 4942 27825

$ cat /proc/meminfo

MemTotal: 31774792 kB
MemFree: 256768 kB
MemAvailable: 15324804 kB
Buffers: 5887324 kB
Cached: 16712984 kB
SwapCached: 488276 kB
Active: 19816476 kB
Inactive: 8835508 kB
Active(anon): 12801324 kB
Inactive(anon): 2522936 kB
Active(file): 7015152 kB
Inactive(file): 6312572 kB
Unevictable: 145612 kB
Mlocked: 48 kB
SwapTotal: 33554428 kB
SwapFree: 28493052 kB
Dirty: 2301336 kB
Writeback: 9564 kB
AnonPages: 6161528 kB
Mapped: 7931416 kB
Shmem: 9272240 kB
KReclaimable: 2201904 kB
Slab: 2451272 kB
SReclaimable: 2201904 kB
SUnreclaim: 249368 kB
KernelStack: 21552 kB
PageTables: 85272 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 49441824 kB
Committed_AS: 30098296 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 55400 kB
VmallocChunk: 0 kB
Percpu: 7712 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 5078656 kB
DirectMap2M: 27346944 kB
DirectMap1G: 0 kB

[Chris Cheney (ccheney)]

I haven't verified if this actually fixes the problem but it seems to make a huge difference in 'free' memory as shown by 'free'.

I had already closed my VMs by the time I tried this so haven't verified if it stops the long freezes yet.

              total used free shared buff/cache available
Mem: 31030 5331 2790 5347 22907 19900
Swap: 32767 3130 29637

echo 3 > /proc/sys/vm/drop_caches

              total used free shared buff/cache available
Mem: 31030 5349 17122 5312 8557 19915
Swap: 32767 3130 29637

Notice the 2790 -> 17122 change above in 'free', and 22907 -> 8557 for buff/cache.

It appears the kernel doesn't get rid of cache in a reasonable timeframe at the expense of swapping programs out to disk while i/o is happening, which causes gui freezes.

[Chris Cheney (ccheney)]

It appears Red Hat fixed this problem with a kernel patch in RHEL 7.5 that was not upstreamed.

https://access.redhat.com/errata/RHSA-2018:1062

BZ - 1471875 - soft lockups during unmount when dentry cache is very large

https://bugzilla.redhat.com/show_bug.cgi?id=1471875

[Chris Cheney (ccheney)]

Actually looking at the patch (shrink_dcache_for_umount) this might not be the whole problem but may be just related. The problem I am having is while the system is still running, and cache growing huge, not while umounting.

[Chris Cheney (ccheney)]

The last time I got it to hang long enough to produce a backtrace it showed:

[1560556.660010] INFO: task vgs:3103183 blocked for more than 1105 seconds.
[1560556.660012] Tainted: G U OE 5.4.0-40-lowlatency #44-Ubuntu
[1560556.660013] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1560556.660014] vgs D 0 3103183 411358 0x80004004
[1560556.660016] Call Trace:
[1560556.660021] __schedule+0x2e5/0x750
[1560556.660023] schedule+0x49/0xd0
[1560556.660024] schedule_timeout+0x21a/0x300
[1560556.660027] ? call_rcu+0x10/0x20
[1560556.660030] ? __percpu_ref_switch_mode+0xec/0x1b0
[1560556.660031] wait_for_completion+0xa3/0x100
[1560556.660033] ? wake_up_q+0x70/0x70
[1560556.660036] exit_aio+0xe7/0xf0
[1560556.660038] mmput+0x22/0x120
[1560556.660039] do_exit+0x2f8/0xae0
[1560556.660041] ? finish_task_switch+0x75/0x250
[1560556.660042] do_group_exit+0x43/0xa0
[1560556.660044] get_signal+0x13e/0x8f0
[1560556.660046] ? hrtimer_cancel+0x15/0x20
[1560556.660047] ? read_events+0x14f/0x160
[1560556.660049] do_signal+0x34/0x6e0
[1560556.660050] ? hrtimer_init_sleeper+0xb0/0xb0
[1560556.660052] ? do_io_getevents+0x7a/0xf0
[1560556.660054] exit_to_usermode_loop+0xbf/0x160
[1560556.660055] do_syscall_64+0x163/0x190
[1560556.660057] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[1560556.660058] RIP: 0033:0x7f3b1269a70d
[1560556.660062] Code: Bad RIP value.
[1560556.660062] RSP: 002b:00007fffd99f2f08 EFLAGS: 00000246 ORIG_RAX: 00000000000000d0
[1560556.660064] RAX: fffffffffffffffc RBX: 00007f3b1228e6c0 RCX: 00007f3b1269a70d
[1560556.660064] RDX: 0000000000000040 RSI: 0000000000000001 RDI: 00007f3b12a7e000
[1560556.660065] RBP: 00007f3b12a7e000 R08: 0000000000000000 R09: 0000000200000016
[1560556.660065] R10: 00007fffd99f2f90 R11: 0000000000000246 R12: 0000000000000001
[1560556.660066] R13: 0000000000000002 R14: 0000000000000040 R15: 00007fffd99f2f90

[JPT (j-p-t)]

@ccheney
Regardless if my crashes are gone, my experience is that having a swap file in recent Linux is deadly. If Linux starts swapping only a little then it's hardly usable any more. I believe this is because of the algorithm that decides which pages to remove from memory.

Your observation with 50% RAM could be related to VMLIM. VMLIM says how much of the RAM may be used for programs. By default this is half your RAM. The other half is for cache and buff only. (as I understood, this may be inaccurate). Try other values for VMLIM.

[JPT (j-p-t)]

@ccheney
better open a new bug because this one accumulates too much info from unrelated problems

[Rumo (sidevoice)]

@ccheney
Do you have any new information on this bug? Did your trick with drop_caches work eventually?
I'm experiencing the same issue for quite some time with both KDE and Gnome (to support the fact that it's DE agnostic) and it definitely seems to occur at times when system starts caching. The GUI becomes completely unresponsive (including the mouse pointer) and the io indicator on laptop blinks violently. Each time this happens I make sure to check and confirm that the amount of swapped data is significant enough (from half a gigabyte to couple of gigabytes)