massive performance issues since 22.04 upgrade

[Henning Sprang]

Hi,
After upgrading to 22.04 i had to fight with massive performance issues.

Browsers appeared to hang every other minute, youtube videos being laggy and hang in between, applications in a virtualbox VM where slow and also hanging every other minute to a level of not being useable. On a pretty recent and powerful system just 2 years old.

I noticed CPU jumps in top, but also somehow thought it could be a graphics issue so invested some time installing nvidia drivers properly.
Also I wondered if it might be the lowlatency kernel I normally use because I do audio stuff, and switched to generic. But nothing helped.

ThenI had the idea it could be a kernel/scheduler issue because the system wasn't always slow, but it appeared certain things kept hanging when other processed had a lot of cpu for a few seconds.

So I got a recent mainline kernel, configured it with my last running config from 21.10 before the update, made the debs and installed them, and now can tell that a mainline kernel 5.17.7 with all the dkms modules that i had before which got compiled automatically at installation brings back a "normal" performance.

I can browse the web, run multiple youtube vids at once, even in another browser, have thunderbird running, and a virtualbox machine open with another browser for some web app testing and everything runs fine and smooth, no lagging.

Not sure yet what the real reason is - either the kernel version, or a patch in the ubuntu version, or the 22.04 kernel config so far, or some configuration made in 21.10 that isn't good with 22.04 and it's kernel anymore.

I will go ahead tomorrow and see if I can build a vanilla kernel with the config from the ubuntu 22.04 kernel and "make oldconfig", then I will be able to tell if only the config is making the difference.

Please let me know of there is anything I should test to further analyze this issue, or any ideas I can try to solve it without having to run a mainline manually installed kernel.

Thanks.

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: linux-image-generic 5.15.0.30.33
ProcVersionSignature: Ubuntu 5.15.0-30.31-generic 5.15.30
Uname: Linux 5.15.0-30-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.20.11-0ubuntu82
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: henning 6198 F.... pulseaudio
 /dev/snd/controlC1: henning 6198 F.... pulseaudio
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Sat May 14 23:02:38 2022
InstallationDate: Installed on 2020-04-12 (761 days ago)
InstallationMedia: Ubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017)
MachineType: LENOVO 20QV00CEGE
ProcFB: 0 i915drmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.15.0-30-generic root=/dev/mapper/vgubuntu-root ro
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-30-generic N/A
 linux-backports-modules-5.15.0-30-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3
SourcePackage: linux
UpgradeStatus: Upgraded to jammy on 2022-04-28 (15 days ago)
dmi.bios.date: 12/06/2021
dmi.bios.release: 1.42
dmi.bios.vendor: LENOVO
dmi.bios.version: N2OET55W (1.42 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20QV00CEGE
dmi.board.vendor: LENOVO
dmi.board.version: SDK0T08861 WIN
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.ec.firmware.release: 1.23
dmi.modalias: dmi:bvnLENOVO:bvrN2OET55W(1.42):bd12/06/2021:br1.42:efr1.23:svnLENOVO:pn20QV00CEGE:pvrThinkPadX1Extreme2nd:rvnLENOVO:rn20QV00CEGE:rvrSDK0T08861WIN:cvnLENOVO:ct10:cvrNone:skuLENOVO_MT_20QV_BU_Think_FM_ThinkPadX1Extreme2nd:
dmi.product.family: ThinkPad X1 Extreme 2nd
dmi.product.name: 20QV00CEGE
dmi.product.sku: LENOVO_MT_20QV_BU_Think_FM_ThinkPad X1 Extreme 2nd
dmi.product.version: ThinkPad X1 Extreme 2nd
dmi.sys.vendor: LENOVO

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This change was made by a bot.

[Matthew Ruffell (mruffell)]

Hi Henning,

Just a thought, could you try booting with IOMMU disabled?

Edit /etc/default/grub, and append "intel_iommu=off" to GRUB_CMDLINE_LINUX_DEFAULT. Then save, and run "sudo update-grub" and reboot.

Between 21.10 and 22.04 we enabled CONFIG_INTEL_IOMMU_DEFAULT_ON=y, and it has caused all sorts of odd issues, including performance related ones, so its worth a shot.

We will be reverting the change to CONFIG_INTEL_IOMMU_DEFAULT_ON, and turning it off again in a kernel update in the near future.

If "intel_iommu=off" doesn't help, then a make oldconfig with the 22.04 config and 5.17.7 would be interesting to see.

Thanks,
Matthew

[Henning Sprang (henning)]

Hi Matthew,

Yes, that looks very good! have to dig a bit deeper and do some more
actual work, but at the first glance I can do the things that I
described above where impossible!

Thanks a lot!

I tried to build 5.17.7 with the 22.04 config and make olddefconfig
and then bindeb-pkg, but it took extremely much longer than with the
20.10 config, and after about 2 hours i saw it still hanging on
dpkg-deb for the dbg package and even half an hour later not finish.

I will try again tonite, probably doing a full "mrproper" before and
letting it run over night to see if it really hangs or just takes
extremely long.
Actually I cant imagine why a dpkg-deb should hang, but I also saw now
clues what it could be.

Cheers,
Henning

[Matthew Ruffell (mruffell)]

Hi Henning,

That is great news that "intel_iommu=off" on the kernel command line did the trick.

You can follow this bug to get updates on when we will be disabling IOMMU by default in the 5.15 kernel. It should be soon, the patches are already submitted, and reviewed by the kernel team. The next step is for the patch to be applied to the git tree and the next kernel update built.

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1971699

I'll let you know once there is a kernel in -proposed with IOMMU disabled. The patches missed this current SRU cycle, and will be in the next one, so it might be in a few weeks / months time.

Thanks,
Matthew

[Henning Sprang (henning)]

I have an update on this and quite a bit of more information, even though not very clear what it means and how to further go on fully fixing it:

After some more usage i realized that the performance is still pretty weak when using my desktop system for slightly more "real" work.

I kept getting stutter in Video playback and Web browsers when using web applications with the slightest bit of JavaScript functionality, and a Virtualbox virtual machine.

I first suspected gnome-shell to be a culprit and had the impressions things where sligthly better when using LXDE as a desktop environment, but still I never got real good performance.

it's a real hard to get a grip on problem, as I dont know how to instantly produce the problem and measure it as with normal bugs.

The easiest thing I can do is open a web browser, got to youtube, and open a few videos in new tabs and try to play them, and skip back and forth to different points, and "pretty soon" I get a stuttering in the video, a mouse pointer not moving fluently, switching tabs taking multiple seconds that usually works in an instant, a browser process with multiple 100s of % of CPU and a RRD* process.

these are things that work without a problem on a very old Lenovo x1 tablet with a mobile processor, and that i would expect to work equally easy on a pretty new X1 extreme with i9 processor and 32G Ram...

I did some more testing ( I cannot call it real debugging as I dont know how to use proper tooling and stuff for these kind of analysis) and checked how other Distributions and other Ubuntu versions perform from USB Live systems and found that not only PopOS, but also Ubuntu 21.10, BUT to my surprise also 22.04 from USB Live system behave much better.

So I thought i must have some setting wrong, but I didn't know which, until i tried to turn off Intel Boost because I remembered in Ubuntu Studio it is recommended to be turned off for good audio performance. This helps a lot!

But, this is not what is different between 22.04 live - because there, Intel boost is ON and it still behaves good.

I tried this with several browsers with always the same result, so it seems to be a generic graphics system/library, or kernel problem. It actually "feels" like some scheduling issue - which was the reason I tried the boost setting.
I also tried with a new test user, to make sure it's not a userspace local setting, or any browser plugin. But the same problems with a new, "clean" user.

The system I experience all this with has been installed first with 20.10, so it has multiple upgrades - it *could* be there is some other setting from previous software versions thats broken - but I have no idea what setting this could be and how I can search for it.

It could also be some old version of some gnome, X11 ( I also tried Xorg as well as wayland, no noticeable difference) or graphics library, and the kernel package is the wrong place for this bug, but I dont know where to start with that then...

[Henning Sprang (henning)]

BTW: also, as I have an nvidia graphics card in that machine, one could think something is wrong with the graphics drivers...
But I tried the nvidia drivers as recommended by the driver GUI tool as well as nouveau. It "feels" like the nvidia drivers are even slightly worse.

On the Ubuntu live USB sticks, where perfromance is better, nouveau is used, on Poop OS the nividia driver - they seem all work similarly good.

[Prudhvi (prudhvi-pk)]

Is there any workaround for this issue, i tried this GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=off", but no improvement in browser performance.

[Matthew Ruffell (mruffell)]

Hi Prudhvi,

We might be chasing a different issue than just IOMMU being on. Do you have the same symptoms as Henning? You might be better off filing a new bug and we can debug your issues specifically.

Henning, are you still building your own kernels? Are things better on the recently released 5.19-rc1? Are you able to try a clean install of 22.04? Did everything work fine under 21.10?

Henning, there is a new kernel in -proposed, 5.15.0-36-generic, where IOMMU is disabled again by default, and it should clear up the IOMMU issues people are having. It should be released around the 20th, give or take a few days if any CVEs turn up.

Thanks,
Matthew

[Henning Sprang (henning)]

@mruffel:
interesting, I am right now already at 5.15.0-37-generic from:
APT-Sources: http://de.archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages

I gave up with custom Kernels as it seemed they provide no clear improvement better then running the ubuntu package with iommu turned of as you recommended.

The last I tried was 5.18.1, but that doesn't work with the virtualbox dkms stuff which I depend on for work. I assume this wont better with 5.19, but i can try later.

I cannot say for sure if / how much of the problem existed in 21.10 because my usage scenario changed around the time when I updated - a new work project, more videocalls, and I am traveling and use the laptop more for media consumption as youtube and videos like this:
https://www.zdf.de/gesellschaft/markus-lanz/markus-lanz-vom-7-juni-2022-100.html

I had twice weekly videocalls with Teams in the browser which drove the CPU up and the fans on, but did not stutter to a point of being impossible.
But now I (try to) use Zoom, every day, and it's basically not possible - I have to use my (twice as old and much weaker mobile CPU) windows tablet for this.

I cannot "easily" start a full clean install as I need the machine for daily work, and setting everything up from scratch takes some while - but I will try to get a second disk to be able to do it but keep the exiting install to work as good as it is. I might be able to restore a pre-update state of my system into a free logical volume and run this. Will see how far i get!

It might very well be that I have multiple problems at once and every one of them contributes a bit to the poor performance I see, also because multiple changes seem to lead to slightly better behaviour each:

* turning iommu off makes things from extremely unsmooth to nearly acceptable
* turning intel_boost off makes things a bit better and acceptable
* moving from gnome to lxde gets me even pretty stutter free video(but not the UX I want...)

It could be a combination of kernel / graphics library / gnome(-shell)...

OR it could be *only* a kernel thing, but slightly suboptimal behaviour of graphics libaries and gnome.

E.g. gnome-shell has a lot of short cpu load peaks on a single core, which should normally not block everything else but maybe can lead to video stuttering when process/IO management and scheduling are wrong in the first place...

I'll keep testing and experimenting when I have spare time and report any findings here...

[Urban Engberg (uengberg)]

For what it is worth, I am experiencing what seems like the exact same issues as Henning, on a Thinkpad T470s after upgrading from 21.10 to 22.04. I am pretty confident there were no problems at all with 21.10, now my whole system and especially things like videos running in chrome gets extremely lagging after the system has been running for a few hours. I tend to do daily reboots just to have the system in a useable state.

I am on 5.15.0-37-generic and have also applied the IOMMU fix without any noticeable effects.

A workaround is strongly needed. Please instruct me if there is any information I can provide from my side/any things to try out.

[Henning Sprang (henning)]

@uengberg:

in my research for another bug/problem (https://bugs.launchpad.net/ubuntu/+source/power-profiles-daemon/+bug/1977660 - I did not yet document it there)
i found out that power profile behaviour is strange and actually i realized that the settings available in gnome *do* have a short term effect on Kernel parameters, but keep jumping back to the other setting.

Then I connected this finding mentally with the finding i made here, namely that performance is better in lxde then in gnome, and I tried to see whats happening if I completely remove the power-profile-daemon, because I though it is probably bad when a Kernel or the Thinkpad firmware cannot properly set power profiles and keepy jumping back and forth...

Wile this is only how I imagine these things working with my less than incomplete knowledge, I can says than Video playback in gnome works quite nicely after having removed the dysfunctional power-profile-daemon package.

You might try to see if that helps you too.
BTW in my case video(and accompanying audio) is/was problematic in all browsers i tested - in firefox, chrome and chromium, as well as in Zoom App(which I guess is likely an electron app). I had no chance to test is zoom works now too.

I still dont know if this all just is a symptomatic remedy or really the root cause, though.

Just as much I dont know if this really is a Kernel, a thinkpad Firmware/ACPI, or another low level library problem.

@mruffel: are there any cpu/performance tests suites we could run to make all this more measurable instead of having to say "oh run a bunch of Youtube videos and/or zoom calls and see if it starts stuttering after a while"? The thing not working properly could be video playback, but it could also just be some scheduling issue and a system that cant cope well with single processes creating short term CPU spikes, as gnome-shell does and as in Video Playback the RDD Process appears to cause.

[Erico Schuch (eschuch)]

Working and testing if the kernel livepatch theory that Kprobe registers as a ftrace handler could be the cause of the problem. But my old I5 is making this process difficult.

[Lucas Davey (eldavey)]

I would like to add that I encountered this issue on Ubuntu 22.04 after a power outage. The conservation mode (power stays at or below 60%) of my Lenovo Legion 5i was then turned off.

[Urban Engberg (uengberg)]

@henning: Thanks for replying to my comment. I actually tried uninstalling the power-profiles-daemon when I saw the message, but running like that for the last month has not made any significant difference to the lag I am experiencing after some hours of use. I have now reinstalled it, just to be sure it doesn't affect performance for me. After a reboot, everything is working smoothly for a while – as always.

I will soon have to find a solution for this issue, will possibly have to do a clean new install. Has anybody got some experience on whether a clean install will solve problems?

[Christopher Hamer (c-edward-hamer)]

I am getting this issue on two machines. One Dell 7530 Precision laptop, I have upgraded 18.04 - 20.04 and 22.04. It is really quite noticeable; after a few hours I start to notice it. After leaving it for a few days it gets worse and worse. Even typing this I suddenly get 15 of the same letter typed and stuttering.

This machines got 6 core Intel i7, 42GB RAM, NVMe SSD. It does have a Radeon GPU but currently booted into the Intel GPU with switchable graphics. Might be worth trying it booted via the Radeon. Might be worth noting it is powered by a Dell Dock which says it doesn't deliver enough power when booting (but it does for my use).

@Urban

I installed 21.10 on a Dell XPS 9365 2 in 1 and upgraded to 22.04. It is happening on that too. Its not anywhere near as noticeable but definitely got some mouse stuttering which 21.10 didn't have.

[Urban Engberg (uengberg)]

@c-edward-hamer: It is interesting that you mention the issue with getting 15 of the same letter while typing. Two days ago I switched to Wayland (from Xorg) to see if that had any impact. There, I had this problem constantly and switched back to Xorg as a consequence, where I don't see this. I believe it must be different symptoms of the same problem, and it definitely seems kernel-related to me...

I am curious why this problem is not affecting more people and getting a top priority?

[mhosken (martin-hosken)]

Just to say I seem to have the same problem as Henning and that iommu=off seems to have helped a lot. I even see my cpu go above 2GHz now!

Lenovo P15s, CPU 1165G7.

[Nuno Sucena Almeida (slug-debian)]

Same problem, two identical machines (CPU, GPU, RAM size, SSD, etc), one with 20.04, the other with 22.04 and the difference is almost night and day. Tried the iommu workaround and didn't seem to make any difference on improving the performance with 22.04

[Michael (tackleberry132)]

Same problem here. Thinkpad P14s Gen1, 32GB Ram. Was working fine on 20.04 - and after the upgrade to 22.04.1, everything is totally laggy, CPU consumption almost always at 100%.

iommu=off made it slightly better, but still not acceptable...

[Sascha H. (sasch12)]

I also have Performance issues, so i managed to get pattern on my machine.
I have a Rycen 5, 32GB Ram and a Nvidia card.

i had 20.04 it was fine, then i did a fresh install for 22.04.
The first 2 days or so i had no trouble, yes it runs a bit laggy but i was fine with it, then i played a game. It was horrible, after closing my browser the game runs fine.
So i startet to search for something, maybe my RAM runs full? Check no it hit 17GB out of 32, so at exactly 17GB it starts to go crazy, CPU usage is ok, some go to like 60% max but it laggs all over the place. I had no CPU usage spike over say 70% on working and/or gaming.
I dont know how to see my GPU usage.

I also can rebuild the youtube Tab lag, after like 12 GB of RAM usage it goes slower, but not much.
I stoped this test at 20GB, i hat a max CPU usage of 84% and ~78% on the rest of the cores.
So, only on my web browser its fine to go to 12GB, at 17GB ist bad at 20GB its the same as 17GB, But a playing a game and the web browser at the same time at 17GB its realy bad.
I dont know maybe someone can work with this informations.

[Kulasangar Gowrisangar (kulasangar)]

I'm also still facing the issue with an 16GB HP laptop. Did a fresh install on the SSD and when I open edge and chrome together it consumes a lot of RAM.

[Peter Sanchez (petersanchez)]

I upgraded from 20.04.2 to 22.04.1 over the weekend using the normal do-release-upgrade process I've always used. Since then I notice Firefox takes 30ish seconds to load. Telegram same. Libreoffice literally 3 minutes to open.

I've tried some of the suggestions in this ticket and the issue remains.

Thinkpad X1 Carbon 6th gen, 16gb ram. Very frustrating bug.

[Hans Deragon (deragon)]

Like everyone else, I am experiencing the same issues. "intel_iommu=off" in Grub does not help.

Here are some symptoms I suffer; not sure if they are related to this bug or not, but worth mentioning since they all degrade my Ubuntu experience since I upgraded from 20.04 to 22.04.

- Overall stuttering; Alt-tab, switch workspace or switch to another application by selecting it and typing lags a few 100ms.
- Gnome animations are not smooth.
- Video stuttering and tearing with any piece of software (VLC, Chrome, Xine...). Have a Skylake GT2 [HD Graphics 520]. Is hardware acceleration enabled? I have no clue how to figure this out.
- gnome-shell always running at more than 6% CPU, jumping often up to 20%, under Xorg. It was way worse when under Wayland. Lets say the average is about 11% CPU.
- System is "frozen" for 5s every time it wakes up from Suspend to RAM (S3). Under 20.04, response was instantaneous. You can move the mouse during that 5s period, but nothing responds to the keyboard or mouse buttons.

This is after an upgrade from 20.04 to 22.04.1. I am running Xorg because when running Wayland, gnome-shell would always be >20% CPU.

My CPU profile is always set at "High Performance" and my CPU clock is always around 2.7x GHz.

System: Dell Latitude E7470 (2016), Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, Skylake GT2 [HD Graphics 520], 32 GiB Ram, 1 TiB SSD, 1900x1080 screen.

[Lu Danin (ludanin)]

I've been having very similar issues on a lenovo legion laptop, disabling intel_iommu helped but doing so wouldn't give me results as smooth as windows/manjaro.

One thing that i noticed is that if my cpu (i7 10750h) gets stressed enough and reach high temperatures Ubuntu will throttle all cores to 800mhz.

Disabling thermald with `sudo systemctl stop thermald && sudo systemctl disable thermald && sudo shutdown -r now` apparently fixed the issue for my laptop. I even removed the `intel_iommu=off` workaround from /etc/default/grub since it seems it's no longer necessary and my system is still super smooth so far

I've been running multiple cpu-heavy docker images from work and the max temperature I recorded in any core was 95C, for reference according to sensors i7-10750h considers any temp up to 100C as high, only above that is that it becomes critical.

Since those images stresses my system a lot (even on manjaro) it's safe to say that even after disabling thermald cpu cooling is still working, otherwise my laptop would've probably shut off by this point

[Urban Engberg (uengberg)]

EDIT: This was the case for almost three weeks in which I didn't reboot the machine, everything was working perfectly. But after rebooting, I'm back with the same issue, having to reboot once every day to get just a reasonable (not perfect) performance. Still no idea what is causing this.

I previously wrote:

Has anybody else noticed some major improvement within the last few days? I am running on kernel version 5.15.0-47-generic and have now been running my laptop for 6 days without needing to reboot, and everything suddenly seems to keep running smoothly. I previously had to reboot every second day to have the machine running anything close to acceptable.

My last reboot was on the 21st, and between the reboot on the 19th and that I can see the following updates in /var/log/dpkg.log:

2022-09-19 10:37:40 status installed gcc-12-base:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:40 status installed libgcc-s1:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:40 status installed libstdc++6:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed libgomp1:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed gcc-12-base:i386 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed libquadmath0:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed libatomic1:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed libgfortran5:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed libubsan1:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed libcc1-0:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed liblsan0:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed libitm1:amd64 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed libgcc-s1:i386 12.1.0-2ubuntu1~22.04
2022-09-19 10:37:41 status installed libc-bin:amd64 2.35-0ubuntu3.1

I have no idea what can have made the difference other than these. The current kernel is from September 1st, so that in itself definitely didn't solve anything.

[francesco (u-user-prop)]

I also noted a high usage of RAM. When in idle with no open apps, I have 4.7 GB of RAM used.

[Trrrmmer (trrrmmer)]

This dist upgrade is a massive clusterfuck.
I disabled turbo and iommu. No change at all in performance.

I regret the upgrade. The old 20.04 worked nicely.

[Lu Danin (ludanin)]

For anyone running into this issue with a modern 10th gen up intel cpu I'd suggest disabling thermald using systemctl and then rebooting

Do it at your own discretion though, but at least on my lenovo laptop my system has been running smoothly with no need to disable turbo boost or iommu

[Hans Deragon (deragon)]

Three days after I commented here (2022-09-26), after and update with reboot and switching back to Wayland, I notice a good performance improvement. I cannot explain why.

I still suffer from Video stuttering and lagging across all players (probably another bug though) and when my system resume from suspend (S3), it is almost "frozen" for 5s at least because the CPU spikes very high for that brief period of time. I cannot figure out what process hugs the CPU like that though.

System: Dell Latitude E7470 (2016), Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, Skylake GT2 [HD Graphics 520], 32 GiB Ram, 1 TiB SSD, 1900x1080 screen.

[Hans Deragon (deragon)]

I just noticed that when I simply move the mouse, gnome-shell CPU jumps up 30% CPU. For those of you suffering from this problem, this is the issue to follow: https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/4130

[kohlerm (markus-kohler-t)]

Had bad performance running in VMware Workstation running on Windows 11 (significant pauses during typing).
Upgraded to
Linux ubuntudev 5.19.17-051917-generic #202210240939 SMP PREEMPT_DYNAMIC Mon Oct 24 09:43:01 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

which seems to fix the problem

[Joel Bacal (joel-bacal)]

Hi to all,

In this tutorial you can find a way to upgrade linux kernel in a not so dificult way: https://linuxhint.com/install-upgrade-latest-kernel-ubuntu-22-04/

In a nutshell:

Step 0: I recommend to do a backup before (using timeshift or other app)

Step 1: sudo apt update

Step 2: sudo add-apt-repository ppa:cappelikan/ppa -y

Step 3: sudo apt install mainline -y

Step 4: Open Ubuntu Mainline Kernel Installer (just search it)

Step 5: Install the kernel you want (I installed the last 6.07)

Step 6: If you disabled grub's timeout reactivate it with grub-customizer (here you can also add intel_iommu=off to GRUB_COMAND_LINE in an ease way), so you can choose the old kernel if things go wrong

I installed 6.07 and added intel_iommu=of, but I took my precautions when using it. I did a complete backup of my system before (you can install timeshift if you don't know how).

I also disabled thermald, uninstalled power-profile-daemon and installed cpu power manager extension from gnome.extensions (https://extensions.gnome.org/extension/945/cpu-power-manager/) to configure it in the higher cpu frequency.

I have a Toshiba C845 with 4 cores i7-3632qm with CPU power manager with top speed 2.9 - 3,2 ghz with 2 displays working and upgraded from 20.04 to 22.04

I saw improvements, but most after enabling again xorg. Seems old generation intel processors are better with it.

Nevertheless, the system is not as fluid as with 20.04 . For me it seems that gnome 42 improved design is more resource hungry than gnome 32.

Seems for my hardware is not worth updating. I also did a backup before upgrading so I think I will return to 20.04.

Hope this help!

[Plamen Dragiyski (plamen-2)]

After updating to 22.04 from 20.04, I have experienced the same performance issues. And adding "inter_iommu=off" did not help. The laptop is with Intel CPU and integrated graphics, so it seemed like the correct option.

Investigating a bit further, I notices unusually low CPU temperature (around 36°C idle, while before the update it was usually around 45°C idle). Installing linux-tools-generic package and using `turboprop` and `stress` to bring all CPU cores to 100%, it showed this:

```
Core CPU Avg_MHz Busy% Bzy_MHz TSC_MHz IPC IRQ SMI POLL C1 C1E C3 C6 C7s C8 C9 C10 POLL% C1% C1E% C3% C6% C7s% C8% C9% C10% CPU%c1 CPU%c3 CPU%c6CPU%c7 CoreTmp PkgTmp GFX%rc6 GFXMHz GFXAMHz Totl%C0 Any%C0 GFX%C0 CPUGFX% Pkg%pc2 Pkg%pc3 Pkg%pc6 Pkg%pc7 Pkg%pc8 Pkg%pc9 Pk%pc10 SYS%LPI PkgWatt CorWatt GFXWatt RAMWatt PKG_% RAM_%
- - 399 99.72 400 1800 0.92 22289 0 0 0 0 0 1 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28 0.00 0.00 0.00 43 43 77.63 300 300 399.82 99.96 20.87 20.84 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.79 0.93 0.12 0.48 0.00 0.00
0 0 399 99.72 400 1800 0.94 1476 0 0 0 0 0 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28 0.00 0.00 0.00 43 43 77.63 300 300 399.83 99.96 20.87 20.84 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.79 0.93 0.12 0.48 0.00 0.00
0 4 399 99.72 400 1800 0.92 1514 0 0 0 0 0 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28
1 1 399 99.72 400 1800 0.90 1692 0 0 0 0 0 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28 0.00 0.00 0.00 43
1 5 399 99.70 400 1800 0.85 11087 0 0 0 0 0 1 0 0 0 0 0.00 0.00 0.00 0.00 0.02 0.00 0.00 0.00 0.00 0.30
2 2 399 99.72 400 1800 0.93 1488 0 0 0 0 0 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28 0.00 0.00 0.00 43
2 6 399 99.72 400 1800 0.94 1335 0 0 0 0 0 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28
3 3 399 99.72 400 1800 0.95 1518 0 0 0 0 0 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28 0.00 0.00 0.00 42
3 7 399 99.72 400 1800 0.96 2179 0 0 0 0 0 0 0 0 0 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28
```

All cores running at 400MHz, which should be the idle cpu-state, not the cpu-state when the cpu is at 100%. Also it must be noted that the slowness remained after reboot and even on the boot screen (after adding intel_iommu=off), including the password screen from the BIOS (outside grub/linux). Restart seems not to help.

However, after going to sleep state (G1 I think), which h...

[Andres Schwartz (anyeos-3)]

I don't know if it is related with what is happening to me, but all my drag actions becomes slow / frozen. It appears that the window is updated internally for each the way the mouse taked (when I am dragging).

It happens only when I have a lot of things opened. Specially if that things uses 3D acceleration (it can be vulkan or OpenGL that does not matter I already tested it).

I monitored the system usage and the CPU is very low usage and the GPU too (3% to 7% CPU and 20% to 33% GPU). But, the system load becomes high (more than 1.0) when I try to drag some window.
When I stop of trying to drag the system load slowly lowers.

I recorded a video with OBS and the recording freezes too in the same moments so It appears to be related with image drawing when using input on a window manager.

It happens too on Budgie Desktop.

The action drains CPU usage (one core almost). I mean, when I drag a window slowy (a terminal with htop running) I can view a very high CPU usage in that moment (And I can ear the CPU fan running faster too).

So it is some function that stay on some loop of some kind related with the drag action only draining CPU instead of waiting and let other tasks to run.

My system information:
intel_iommu=off
Linux 5.15.0-52-generic
NVIDIA 520.56.06
Ubuntu 22.04.1 LTS

[fINTiP (lacking-latin)]

Just want to say, I had this same issue, and disabling hardware acceleration in the discord desktop app's `settings -> advanced` fixed it for me. dragging any windows was horribly laggy.

[fINTiP (lacking-latin)]

(nvidia 515, pop os 22.04)

[cotigao (vikram2784)]

thinkpad p15: all browsers, vscode and other UI apps are terribly slow.I tried both intel graphics and nvidia graphics. Youtube playback is jittery. Overall ubuntu 22' performance is sluggish!

[[e-mail] (faranta)]

I came to find a bug that I logged that was very similar to this one, to mark it as linked - https://bugs.launchpad.net/ubuntu/+bug/1982560 - but it's gone! Can't see any history on this. Not even closed, just disappeared. Where do I see deleted bugs please?

[chris (platoscave)]

I'm on a Dell XPS laptop and had many of the problems described above including
- one of the cpu's peaked every few seconds
- video freeze
- intermittent bluetooth mouse/keyboard
- repeating characters when typing
- slow browser tab switching

sudo apt update && apt upgrade -y

followed by reboot fixed the problem for me.

[Hans Deragon (deragon)]

I conquer with chris (platoscave). Pretty all my problems went a away with time by magic. It is obvious that many bugs plagued us and where fixed by the different projects composing Ubuntu as time passed by. 22.04 just wasn't ready went it came out.

My only issue that remains is when waking up my laptop from S3, everything is (almost) frozen for at least 5 seconds; seams that the CPU hits 100% (no confirmation) for some reason. But that should be another bug report. I am running under Wayland now, btw.

System: Dell Latitude E7470 (2016), Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz, Skylake GT2 [HD Graphics 520], 32 GiB Ram, 1 TiB SSD, 1900x1080 screen.

[JamesA (ja-101705)]

Hi All,

I too have been experiencing a laggy Ubuntu 22.04.2 on Virtual Box since upgrading from a previous version. For me I don't normally use the web browser much as I use my host pcs browser for that, so I cant speak for how Ubuntu's web browser's performance was before or after upgrading to 22.04. Most of my issues were using the terminal and running gradle tasks which would just pause or sit there and do nothing for a long time before proceeding. In addition just typing in the terminal was extremely laggy. I tried all the usual suggestions for the Virtual Box settings under the System Tab and the Display tabs but no luck. I also tried the suggestion of "intel_iommu=off" with no luck. It wasn't until I did the "sudo systemctl stop thermald && sudo systemctl disable thermald && sudo shutdown -r now" that my performance issues seem to be resolved.

I was hoping since most of this threads content was from 2022, that by now my system would be magically fixed with the updates since then but that wasn't the case. Should this issue have been resolved by now like I am thinking with the normal updates that Im getting? Thanks in advance!

[JamesA (ja-101705)]

Well my results from my previous post seems to be short lived as I still seem to be stuck with a slow system from the console perspective with running gradle tasks. Not sure what to try next.

[Jayan Varma (jayantsr)]

Hi

I have the same issue even with my mini desktop's wired internet and wifi. Today I have tried Ubuntu 23.04 and that too has the same problems. This bug is not mitigated on Ubuntu 23.04 too I believe.

I have to go to Windows 11 which the system came with since I cannot afford the slowness in my work.

[Benico van der Westhuizen (benicovdw)]

update 2023-12-30 ... disabled hyper threading in the BIOS ... machine is MUCH faster ... problem is gone (so far) kernel 6.5.0-1009-oem (tlp not installed ... laptop-mode / tools installed)

Dell Latitude E6540 (2014) ... Intel i7-4600M
Problem seems related to new after market battery, load shedding when the power comes on and off (also when I start the generator, etc.) The state of tlp changes from AC to battery and it also affects the governor. It seems that all the detail for the battery is not properly detected when the battery is 100% charged (tlp-stat -b shows the capacity as not available ... but when it is at less than 100% it displays it correctly.)

I tried to disable the Intel "C States" in the BIOS as well as /etc/default/grub: GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=0"
but the fix did not last.

I watch Frequencies with: watch -n 0.3 "cat /proc/cpuinfo | grep MHz"
fwiiw
I also apt purged: linux-tools-* cpufreq* gamemode*

So what does currently work is (when the battery is not at 100%) then I use the underneath script to play around and that seems to fix the cpu scaling (on kernel 6.2)

Faster way to try and tune stuff:
su -
./manage_governor.sh
#================================================================================
#!/bin/bash

reserv () {
 systemctl restart acpid.service ;
 systemctl restart lm-sensors.service ;
 systemctl restart machine.slice ;
 systemctl restart power-profiles-daemon.service ;
 systemctl restart gpu-manager.service ;
 systemctl restart powertop.service ;
 systemctl restart thermald.service ;
 systemctl restart tuned.service ;
 systemctl restart tlp.service ;
 upower -d ;
 tlp recalibrate ;
 tlp-stat -b ;
}

GOV=`cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor` ;

govdrv () {
 echo "current driver: ";
 grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_driver ;
 }

govtuning () {
 echo "current tuning parameters of $GOV governor:
grep . /sys/devices/system/cpu/cpufreq/$GOV/* " ;
 grep . /sys/devices/system/cpu/cpufreq/$GOV/* ;
 }

getgov () {
 echo "current governor: ";
 grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor ;
 echo "available governors: " ;
 cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors ;
 }

setgov () {
 getgov ;
 read -p " watter governor wil jy apply ? ... " GOV ;
 echo $GOV > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor ;
 echo $GOV > /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor ;
 echo $GOV > /sys/devices/system/cpu/cpu2/cpufreq/scaling_governor ;
 echo $GOV > /sys/devices/system/cpu/cpu3/cpufreq/scaling_governor ;
 getgov ;
 }

watchfreq () {
 watch -n 0.3 "cat /proc/cpuinfo |grep MHz" ;
 }

tunegov () {
 echo "=====================================================" ;
 grep . /sys/devices/system/cpu/cpufreq/conservative/up_threshold ;
 echo "setting conservative governor up_threshold to 40%: " ;
 echo -n 40 > /sys/devices/system/cpu/cpufreq/conservative/up_threshold ;
 grep . /sys/devices/system/cpu/cpufreq/conservative/up_threshold ;

 echo "=====================================================" ;
 grep . /sys/devices/system/cpu/cpufreq/conservative/sampling_down_...

[Shashank Chintalagiri (shashank-chintalagiri)]

This problem still very much exists on 23.04. Since I can't find a single 'regular' metric that shows the problem, I can only provide anecdotal information. Perhaps someone can see a pattern and find a common thread.

The machine I'm using is an Asus Zephyrus G14 with AMD Ryzen 5 4600HS NVIDIA GeForce GTX 1650 Ti and using Nvidia proprietary drivers. Storage is entirely NVME SSD and I have 40 gigs of RAM. I use both KDE (preferred) and Gnome, depending on which feels less laggy at the time. I use X11 now because Nvidia requires it (?), and I remember using wayland some time ago (on the same machine and with nvidia drivers. I remember needing to switch away from firefox because it didn't work with wayland (?).

I did notice significant performance degradation when I updated to 22.10. At the time, I uninstalled most snaps except the browser(s), completely disabled tracker-miner, and the system was more or less usable. In retrospect, I have had performance degradation over the past couple of years when playing Dota 2. Since that's an occasional thing for me, and given gaming on Linux has been historically difficult, I just chalked it up to the game becoming more demanding. Similarly with pycharm and browsers, since I am used to having a large number of files / tabs / projects open, I have subconsiously adapted by changing my behavior by reducing the number of open windows / browser tabs and occasionally exiting or restarting the machine. For context, I've been using Ubuntu since 2007 and it wasn't unusual for my laptops to 20+ days of uptime.

The performance degradation became entirely debilitating after upgrading to 23.04, which I did to get rid of the annoying EOL message for 22.10 (on a side note, 9 months of support on a release? Really?).

The following are the 'measurables' I refer to when I talk of performance degradation:
 - The time it takes for me to hit a key on the keyboard and the character to show up on screen.
 - The time it takes for zsh/oh-my-zsh to render the first and subsequent command prompt.
 - The level of detail I can see when a web page is being rendered.
 - The time it takes to switch focus
 - The time it takes for me to press alt+tab or meta and be able to switch windows.
 - The time it takes for pycharm autocomplete to do something

All of these measurables were probably actually borderline measurable immediately after the update. As in, with a stopwatch and human response times.

I have made the following observations / changes since (possibly in slightly difference sequence). Each of these steps provided enough of an improvement that they felt successful:

  - Disabled most zsh/oh-my-zsh plugins for shell enrichment. Along with that, disabled keychain and its counterpart for gpg.
  - Got rid of snap. Completely. Every snap, and snapd itself as well. I will be honest and say I don't remember why I decided snap was the problem. Doing this did improve the situation slightly.
  - Suppressed tracker-miner. This was doing a lot of disk IO and eating CPU as well.
  - Uninstalled plocate and the indexer. Was also doing a lot of disk IO.
  - Installed (and later uninstalled) preload. I'm less certain if it helped, bu...

[Shashank Chintalagiri (shashank-chintalagiri)]

On further digging around, I noticed the nvidia temperatures were rising and utilization was in the 30-40% range when I was doing things (firefox, terminal, pycharm).

I then remembered that while I had originally configured graphics to use nvidia only on demand around 20.04, this had reverted to nvidia by default to everything on some update. I didn't really try to fix it since it was sort of working and trying to get prime and bumblebee working have eaten up so much of my life already.

I have now switched the PRIME profile in nvidia-settings to ondemand. This has improved performance significantly. I have created the prime-run script from https://askubuntu.com/questions/1364762/prime-run-command-not-found to run specific applications with prime, and it seems to be working.

I am still using x11.

[Shashank Chintalagiri (shashank-chintalagiri)]

I can confirm that switching nvidia to on-demand only has resolved all of the performance issues.

nvidia being used though prime-run script linked above seems to be working. It is running hotter than I would like, but I have no reason to believe that is unexpected. There are some glitching issues that occur in this setup, specifically, the aspect ratio gets skewed approximately 50% of the time any application is launched in full screen mode.

When nvidia is set to be the primary GPU, overall system performance is in the toilet.
   - This is unrelated to running any obviously GPU intensive workloads.
   - It seems to occur on both Gnome and KDE (both with X11). Have not tested with any other DEs.
   - Browsers seem to be exacerbate the problem. A lot. Chromium engine degrades worse than Gecko.
   - The problem manifests most obviously in workloads which are IO bound. For example, zsh tab completion.
   - CPU utilization via htop is elevated but far from saturated.
   - IO bound processes are slow and show up on iotop. Once these processes (ex. tracker-miner, etc.) are killed, suppressed, or uninstalled, the problem persists and there is no noticeable elevation in io rates.
   - GPU utilization is never saturated. 10-40% on a 1650 Ti is the most I have seen it in non-GPU intensive workloads, even when the system is nearly unusable around it..

[Roland Lewis (roland-l)]

I've seen many of the OP's symptoms too, including pinning one CPU at 100%. Makes the machine unusable. However I can be a little more precise as to when they started:

- 20.04 was fine at first;
- 20.04 later under kernel 5.13-generic saw the problems start, at which pointed I switched to low-latency in desperation;
- 20.04 kernel 5.13-low-latency, which (unlike generic) still worked fine;
- 20.04 under kernel 5.15-generic and 5.15-low-latency were both problematic;
   ... then upgraded to 22.04 to see if resolved;
- 22.04 kernel 6.2 is still problematic.

My guess is a kernel config change which was brought in first for 5.13-generic, and then also introduced to low-latency a version later. I attach diffs for the configs for (a) 5.11-5.13-generic, and for (b) 5.13-5.15-low-latency. I've tried to isolate a changed variable which is common to both, though I have not been able to yet.

Any help appreciated. intel_iommu=off doesn't help.

I'm using a Mac Pro 1,1 with clean EFI boot (and 'noefi' grub params). Woodcrest Xeon X5160 and AMD 5770 (official Mac edition).

[Benico van der Westhuizen (benicovdw)]

Installed:
https://github.com/AdnanHodzic/auto-cpufreq
And sometimes it works and somtimes not.

using the ap fucntion with the script below helps sometimes.
https://github.com/benicovdw/manage-cpufreq/blob/main/manage_governor.sh

[Roland Lewis (roland-l)]

I found a solution for this, at least for my case! I traced the CPU pinning to ACPI GPE11 (corresponding to IRQ9 in my Mac Pro 1,1) being hit constantly.

This can be killed permanently by passing "acpi_mask_gpe=0x11" to the kernel at boot.

I still believe this was first triggered for 20.04 first during the generic 5.11-5.13 transition and then later during the low-latency 5.13-5.15 transition using the official Ubuntu kernels. I still haven't been able to pinpoint if this was a config change's fault, or perhaps whether a microcode error crept in.

Suggest people give this a go. It clearly works for some others. For more background see here:
https://unix.stackexchange.com/questions/588018/kworker-thread-kacpid-notify-kacpid-hogging-60-70-of-cpu

Also a couple of posts here:
https://mattgadient.com/linux-dvd-images-and-how-to-for-32-bit-efi-macs-late-2006-models/