Poor desktop frame rate in Nvidia Wayland sessions

[Daniel van Vugt]

On a desktop with a single Nvidia GPU and single monitor, desktop performance is poor in Wayland sessions.

The main culprit seems to be that all rendering gets deferred until meta_drm_buffer_gbm_new_lock_front by the Nvidia driver, which blocks the main loop at a time we are not optimised to handle blocking.

ProblemType: Bug
DistroRelease: Ubuntu 24.10
Package: nvidia-driver-560 560.35.03-0ubuntu3
ProcVersionSignature: Ubuntu 6.11.0-7.7-generic 6.11.0-rc7
Uname: Linux 6.11.0-7-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.30.0-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: pass
Date: Thu Sep 19 14:13:32 2024
InstallationDate: Installed on 2024-06-12 (99 days ago)
InstallationMedia: Ubuntu 24.10 "Oracular Oriole" - Daily amd64 (20240528)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
SourcePackage: nvidia-graphics-drivers-560
UpgradeStatus: No upgrade log present (probably fresh install)

[Daniel van Vugt (vanvugt)]

Any ideas folks?

[Daniel van Vugt (vanvugt)]

Also keep an eye on https://www.reddit.com/r/pop_os/comments/1fjufso/nvidia_560_drivers_broke_my_system/

[Jose Ogando Justo (joseogando)]

It would be good to have a comparison with https://www.nvidia.com/en-us/drivers/unix/

I would like to see if the lack of Power Profiles, as you mentioned, is a direct result of packaging or if it comes from the driver itself.

Can you try that?

[Daniel van Vugt (vanvugt)]

Same issue in 535. Is it the new kernel perhaps?

[Daniel van Vugt (vanvugt)]

Yes it's the kernel. Reverting to kernel 6.8 has fixed the performance issue on 535. I am yet to test 6.8 with 560...

And it appears the missing information in NVIDIA Settings is still missing, so that's unrelated to the performance regression.

[Alessandro Astone (aleasto)]

> In 'NVIDIA Settings' it looks like the reason might be a lack of power profiles ("Performance Levels").

I don't see those in the 550 driver either though.

Remembering that 560 is the first driver we ship with Wayland explicit-sync support, I wonder if __NV_DISABLE_EXPLICIT_SYNC=1 changes the behaviour.

[Alessandro Astone (aleasto)]

> Same issue in 535. Is it the new kernel perhaps?

Oh. Okay then...

[Daniel van Vugt (vanvugt)]

These results are weirdly consistent and repeatable across purges, reinstalls and reboots:

SLOW: Nvidia-560 + Linux 6.8
SLOW: Nvidia-560 + Linux 6.11
SLOW: Nvidia-535 + Linux 6.11
FAST: Nvidia-535 + Linux 6.8
FAST: Nvidia-550 + Linux 6.11

I'm currently testing on a Turing card, if that's important.

[Daniel van Vugt (vanvugt)]

GNOME 46.5 seems to have the same issue as 47.0 so I don't think it's a Mutter regression.

[Alessandro Astone (aleasto)]

I get good performance with Nvidia 560 on both kernel 6.11.0 and 6.8.0.
This was a fresh install of Oracular, using a Pascal card.

[Maia Everett (linneris)]

I don't know if it's related, but after an update that landed between Sunday and Monday, I'm experiencing a number of graphical issues in Oracular with all tested versions of the NVIDIA proprietary drivers.

I did a clean reinstall of Oracular, and here are the results:

nouveau:
- Everything works, but there's no sound (I'm using the speakers of my monitor, connected via DisplayPort)

nvidia 560:
- Plasma X11 = loading screen spins for a while, then black screen with cursor
- Plasma Wayland = black screen with cursor
- GNOME X11 = blank wallpaper
- GNOME Wayland = the desktop works, but apps using GPU acceleration are bugged, failing to launch or working slowly

nvidia 550:
- Plasma X11 = works fine
- Plasma Wayland = black screen with cursor

Kernel 6.11.0-7-generic. On Sunday (September 15), everything worked.

[Daniel van Vugt (vanvugt)]

> I get good performance with Nvidia 560 on both kernel 6.11.0 and 6.8.0.

The difference I'm seeing is small enough that I estimate it would be completely hidden on a better card than the GTX 1650 I'm usually testing with.

[Daniel van Vugt (vanvugt)]

Re comment #10: Mutter could potentially still get a fix, even if it's needed in other releases too.

[Daniel van Vugt (vanvugt)]

Using kernel 6.11 I consistently get results like these from glmark2-wayland:

  nouveau...: ~1000 FPS
  nvidia-550: ~8000 FPS
  nvidia-560: ~2000-4000 FPS

but I can solve most of it with env __NV_DISABLE_EXPLICIT_SYNC=1 glmark2-wayland:

  nvidia-560: ~7000 FPS

Though applying the same variable system-wide didn't fix desktop stutter. So that's a separate bug. Real time profile results suggest my desktop stutter is gnome-shell spending most (60%) of its time inside lock_front_buffer (also mentioned in https://gitlab.gnome.org/GNOME/mutter/-/issues/3461).

TL;DR - I think there are two or three bugs here:

1. nvidia-560 is much slower than nvidia-550 in glmark2-wayland unless you set __NV_DISABLE_EXPLICIT_SYNC=1

2. nvidia-560 spends way too much of gnome-shell's real time (60%) inside gbm_surface_lock_front_buffer (called by lock_front_buffer).

3. nvidia-535 erratic results with different kernels (comment #9). Let's ignore that for now.

[Daniel van Vugt (vanvugt)]

Actually I don't think this is unique to 560:

> 2. nvidia-560 spends way too much of gnome-shell's real time (60%) inside gbm_surface_lock_front_buffer (called by lock_front_buffer).

It looks like 550 was doing the same.

But gnome-shell on nvidia-550 did consistently profile differently to 560. In nvidia-550 20-25% of the main loop wakeups went to repaints. In nvidia-560 around 5% of main loop wakeups went to repaints. So it sounds like 560 is somehow spinning the main loop too much.

[Daniel van Vugt (vanvugt)]

I can't reproduce the explanation in comment #16 for nvidia-550 being faster than nvidia-560 (at least on Turing). Though it sounds like [1], that doesn't seem to affect the shell performance itself.

We've verified that most of the shell's performance problem comes down to an architectural difference between Mesa and Nvidia drivers [2]. And while I thought triple buffering should work around that, it doesn't yet [3].

So [2] is the main fix that the Nvidia driver needs (not counting subtle power saving/scaling differences between 550 and 560). And [3] is a performance improvement to triple buffering that might be possible, and would work around the need for changes to the proprietary Nvidia driver.

For completeness there is also benchmark performance [1], which is probably something most Ubuntu users won't notice in everyday usage.

[1] https://gitlab.gnome.org/GNOME/mutter/-/issues/3720
[2] https://gitlab.gnome.org/GNOME/mutter/-/issues/3461#note_2236305
[3] https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1441#note_2236368

[Daniel van Vugt (vanvugt)]

Alright we can ignore the benchmark results [1].

And let's not focus on 550-560 performance differences anymore. We know the main performance problem in 560 by a long way and it will either require a new driver (that is probably not "560") [2], or a redesign of triple buffering [3].

So this is now triaged and I have a plan for GNOME 48.

[1] https://gitlab.gnome.org/GNOME/mutter/-/issues/3720
[2] https://gitlab.gnome.org/GNOME/mutter/-/issues/3461#note_2236305
[3] https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1441#note_2236368

[Alexandre Hen (atropine07)]

Hi !

I saw a positive difference in performance for my desktop PC (RTX 2080) on 24.10 by switching to the Nvidia proprietary 560 driver (not the open one) and disabling GSP by adding "nvidia.NVreg_EnableGpuFirmware=0" in "GRUB_CMDLINE_LINUX" and updating Grub.

Kernel is the lastest from the Ubuntu repo.

Thank you,

[Luis Alvarado (luisalvarado)]

Was going to post this today. I am currently testing 25.04 with the 570 driver. On X it works perfectly and all games run beautifully. But when I changed to wayland, there was huge input lag, latency and even freezing when on the desktop (It worked inside the game, although the game lost a chunk of FPS on wayland versus X). Is there a way to grab information about why wayland would freeze (and unfreeze if I ALT+TAB between apps, to then freeze again later).

My test was because HDR came out, so I see it shows when using wayland but it is not good for typical workflow or gaming yet based on what I just went through.

[Daniel van Vugt (vanvugt)]

"huge input lag, latency and even freezing" sounds different to this bug. I suggest opening a new bug about that.

I'll reword this one to make it more clear what it's about.

[Daniel van Vugt (vanvugt)]

It's starting to feel like this is fixed (well enough) in 570.133.07.

[Daniel van Vugt (vanvugt)]

Also tracking in: https://github.com/NVIDIA/open-gpu-kernel-modules/issues/743

[Alessandro Astone (aleasto)]

Daniel, should this still be mentioned in the "Known Issues" section of the Release Notes for 25.04?

[Daniel van Vugt (vanvugt)]

No, I've updated the release notes now.

[Daniel van Vugt (vanvugt)]

I'm not sure upstream bug https://gitlab.gnome.org/GNOME/mutter/-/issues/3461 belongs here. If you go by its title then no. If you go by the 'lock_front_buffer' discussion and the fact that the upstream bug is still open then yes.

If you go by the title of https://gitlab.gnome.org/GNOME/mutter/-/issues/3461 alone then it would have belonged to bug 2064205 or bug 2105500, and should be closed already. So I suspect the upstream bug's title needs changing now.