Slow refresh rate (5-6hz) on Display managed by gnome-shell

[beadon]

NVIDIA GPU sending out frames to an LG Electronics 27" display.

At the moment, just a single display.

The NVIDIA GPU is available across a Thunderbolt 3 link.

Here is what the Nvidia Drivers are reporting:

beadon@semiauto:~$ nvidia-smi
Mon Jul 10 20:35:25 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce GTX 1650 Off | 00000000:52:00.0 On | N/A |
| 0% 33C P8 N/A / 85W | 99MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 2341 G /usr/bin/gnome-shell 1MiB |
+---------------------------------------------------------------------------------------+

Performing an strace I am seeing that the gnome-shell is **very** frequently hitting :
EAGAIN (Resource temporarily unavailable)

It's not clear to me what is causing this. strace on the 33 threads included for reference. To show different behaviors, clicks, movement of windows, typing in a terminal, as running glxgears to show(get) the refresh rate of 5 frames per second.

Is there some kind of heavy CPU-based communication happening for gnome-shell ? I am seeing the CPU usage spike to 90+% when simple movements are done on screen. However, the nvidia-smi tools reports that it truly is attached to the right GPU, the PID is correct.

Please see the strace for further information.

ProblemType: Bug
DistroRelease: Ubuntu 23.04
Package: gnome-shell 44.2-0ubuntu1
ProcVersionSignature: Ubuntu 6.2.0-24.24-generic 6.2.12
Uname: Linux 6.2.0-24-generic x86_64
NonfreeKernelModules: nvidia_modeset nvidia
ApportVersion: 2.26.1-0ubuntu2
Architecture: amd64
CasperMD5CheckResult: pass
CurrentDesktop: ubuntu:GNOME
Date: Mon Jul 10 20:40:20 2023
DisplayManager: gdm3
InstallationDate: Installed on 2023-01-09 (182 days ago)
InstallationMedia: Ubuntu 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809.1)
ProcEnviron:
 LANG=en_US.UTF-8
 PATH=(custom, no user)
 SHELL=/bin/bash
 TERM=xterm-256color
 XDG_RUNTIME_DIR=<set>
RelatedPackageVersions: mutter-common 44.2-0ubuntu1
SourcePackage: gnome-shell
UpgradeStatus: Upgraded to lunar on 2023-04-26 (75 days ago)

[Daniel van Vugt (vanvugt)]

Your log is mostly showing bug 1967707 but I think the root cause that comes before it is:

[ 199.493073] semiauto gnome-shell[2341]: meta_frame_native_release: assertion '!frame_native->kms_update' failed

[Daniel van Vugt (vanvugt)]

To work around this problem, please log into a Xorg session instead.

[Daniel van Vugt (vanvugt)]

Actually I think I've seen this before. One of the graphics drivers has run out of memory:

[ 198.184944] semiauto gnome-shell[2341]: Failed to allocate onscreen framebuffer for /dev/dri/card1: Failed to create dumb drm buffer: Cannot allocate memory
[ 198.209428] semiauto gnome-shell[2341]: Failed to allocate onscreen framebuffer for /dev/dri/card1: Failed to create dumb drm buffer: Cannot allocate memory
...
[ 199.493073] semiauto gnome-shell[2341]: meta_frame_native_release: assertion '!frame_native->kms_update' failed
[ 199.494175] semiauto gnome-shell[2341]: (../clutter/clutter/clutter-frame-clock.c:425):clutter_frame_clock_notify_presented: code should not be reached
[ 199.494260] semiauto gnome-shell[2341]: (../clutter/clutter/clutter-frame-clock.c:425):clutter_frame_clock_notify_presented: code should not be reached
[ 199.527787] semiauto gnome-shell[2341]: (../clutter/clutter/clutter-frame-clock.c:425):clutter_frame_clock_notify_presented: code should not be reached

If I recall correctly the solution is to just reboot. But certainly gnome-shell shouldn't keep trying to run in this situation.

[Daniel van Vugt (vanvugt)]

And I suspect the "5-6hz" is just that it managed to allocate some dumb buffers, but not enough of them to get every frame to the monitor.

[Daniel van Vugt (vanvugt)]

To avoid all of these problems, please just select 'Ubuntu on Xorg' at the login screen.

[Daniel van Vugt (vanvugt)]

Perhaps I over-thought that. Since you actually get some frames on screen and they're just slow, I think the real issue is bug 1970291:

> Secondary GPU initialization failed (Failed to create gbm_surface: Function not implemented). Falling back to GPU-less mode instead, so the secondary monitor may be slow to update.

As for the flooded log, that's bug 1967707. But it sounds like that flood of log messages for you was actually from a different time (bug 2026789) where the screen froze instead of being just slow.

[beadon (bryant-eadon)]

I was unable to select the "Ubuntu on Xorg" as a configuration option, so I did it manually.

I modified /etc/gdm3/custom.conf

current configuration:

# GDM configuration storage
#
# See /usr/share/gdm/gdm.schemas for a list of available options.

[daemon]
# Uncomment the line below to force the login screen to use Xorg
WaylandEnable=false

# Enabling automatic login
# AutomaticLoginEnable = true
# AutomaticLogin = user1

# Enabling timed login
# TimedLoginEnable = true
# TimedLogin = user1
# TimedLoginDelay = 10

[security]
DisallowTCP=false

[xdmcp]
ServerArguments=-listen tcp

[chooser]

[debug]
# Uncomment the line below to turn on debugging
# More verbose logs
# Additionally lets the X server dump core if it crashes
Enable=true

This does not appear to work-around the problem. I am getting drm errors :

[Tue Jul 11 13:24:20 2023] input: Razer Razer Core X Chroma Keyboard as /devices/pci0000:00/0000:00:07.2/0000:50:00.0/0000:51:04.0/0000:53:00.0/0000:54:02.0/0000:57:00.0/usb9/9-2/9-2:1.1/0003:1532:0F1A.000C/input/input39
[Tue Jul 11 13:24:20 2023] [drm:nv_drm_dumb_map_offset [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00005200] Failed to lookup gem object for mapping: 0x00000006
[Tue Jul 11 13:24:20 2023] [drm] [nvidia-drm] [GPU ID 0x00005200] Framebuffer memory not appropriate for scanout
[Tue Jul 11 13:24:20 2023] [drm:nv_drm_dumb_map_offset [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00005200] Failed to lookup gem object for mapping: 0x00000007
[Tue Jul 11 13:24:20 2023] [drm] [nvidia-drm] [GPU ID 0x00005200] Framebuffer memory not appropriate for scanout
[Tue Jul 11 13:24:20 2023] [drm:nv_drm_dumb_map_offset [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00005200] Failed to lookup gem object for mapping: 0x00000008
[Tue Jul 11 13:24:20 2023] [drm] [nvidia-drm] [GPU ID 0x00005200] Framebuffer memory not appropriate for scanout
[Tue Jul 11 13:24:20 2023] [drm:nv_drm_dumb_map_offset [nvidia_drm]] *ERROR* [nvidia-drm] [GPU ID 0x00005200] Failed to lookup gem object for mapping: 0x00000009
[Tue Jul 11 13:24:20 2023] [drm] [nvidia-drm] [GPU ID 0x00005200] Framebuffer memory not appropriate for scanout
[Tue Jul 11 13:24:20 2023] input: Razer Razer Core X Chroma as /devices/pci0000:00/0000:00:07.2/0000:50:00.0/0000:51:04.0/0000:53:00.0/0000:54:02.0/0000:57:00.0/usb9/9-2/9-2:1.1/0003:1532:0F1A.000C/input/input40
[Tue Jul 11 13:24:20 2023] hid-generic 0003:1532:0F1A.000C: input,hidraw8: USB HID v1.11 Keyboard [Razer Razer Core X Chroma] on usb-0000:57:00.0-2/input1
[Tue Jul 11 13:24:20 2023] input: Razer Razer Core X Chroma as /devices/pci0000:00/0000:00:07.2/0000:50:00.0/0000:51:04.0/0000:53:00.0/0000:54:02.0/0000:57:00.0/usb9/9-2/9-2:1.2/0003:1532:0F1A.000D/input/input41
[Tue Jul 11 13:24:20 2023] hid-generic 0003:1532:0F1A.000D: input,hidraw9: USB HID v1.11 Mouse [Razer Razer Core X Chroma] on usb-0000:57:00.0-2/input2
[Tue Jul 11 13:24:21 2023] ax88179_178a 10-1:1.0 eth0: register 'ax88179_178a' at usb-0000:57:00.0-1, ASIX AX88179 USB 3.0 Gigabit Ethernet, 98:bb:1e:1e:f9:7c
[Tue Jul 11 13:24:21 2023] usbcore: registered new interface driver ax88179_178a
[Tue Jul 11 13:24:21 2023] ax88179_178a 10-1:1.0 enx98bb1e1ef97c: renamed from e...

[beadon (bryant-eadon)]

As expected I am using Xorg now,

beadon@semiauto:~$ echo $XDG_SESSION_TYPE
x11

[beadon (bryant-eadon)]

Notably -- there seems to be a known issue with DisplayPort connectivity 1.3 and 1.4 and NVIDIA GPUs recently.

I am seeing a firmware update issued by NVIDIA on June 26 2023 for SOME NVIDIA chipsets:
https://www.nvidia.com/en-us/drivers/nv-uefi-update-x64/

The GTX 1650 chipset does not appear to be impacted.

[beadon (bryant-eadon)]

worse still , now I cannot get a gdm login unless I disconnect the USB-C eGPU (this NVIDIA GPU enclosure).

I am reading up further on "prime" and "reverse prime" GPUs. The behavior here may be similar:
https://forums.developer.nvidia.com/t/failed-to-lookup-gem-object-for-mapping/186526

Here is a log of the connect/disconnect (see attached log).

[beadon (bryant-eadon)]

I am also seeing a couple things like this ..

options nvidia-drm modeset=1

Perhaps this is a path I need to take and specify PRIME vs non-prime ?
https://forums.developer.nvidia.com/t/prime-and-prime-synchronization/44423

[beadon (bryant-eadon)]

ok, SUPER interesting -- when I force the intel profile to be PRIME, then I can safely boot with the enclosure attached(!), the systems gets through gdm, and I can login no problem.

I suspect there is a driver problem we are chasing, and PRIME was just selecting the NVIDIA GPU for some reason and flailing ..'

Hmm running into a bit of a snag :

beadon@semiauto:~$ sudo nvidia-settings
[sudo] password for beadon:

ERROR: NVIDIA driver is not loaded

(nvidia-settings:3802): GLib-GObject-CRITICAL **: 14:38:11.591: g_object_unref: assertion 'G_IS_OBJECT (object)' failed

** (nvidia-settings:3802): CRITICAL **: 14:38:11.592: ctk_powermode_new: assertion '(ctrl_target != NULL) && (ctrl_target->h != NULL)' failed

ERROR: nvidia-settings could not find the registry key file or the X server is not accessible.
       This file should have been installed along with this driver at
       /usr/share/nvidia/nvidia-application-profiles-key-documentation. The application
       profiles will continue to work, but values cannot be prepopulated or validated, and
       will not be listed in the help text. Please see the README for possible values and
       descriptions.

** Message: 14:38:11.616: PRIME: Requires offloading
** Message: 14:38:11.616: PRIME: is it supported? yes
** Message: 14:38:11.637: PRIME: Usage: /usr/bin/prime-select nvidia|intel|on-demand|query
** Message: 14:38:11.637: PRIME: on-demand mode: "1"
** Message: 14:38:11.637: PRIME: is "on-demand" mode supported? yes

[beadon (bryant-eadon)]

Then onto check why the GPU is not even being detected ...

beadon@semiauto:~$ sudo nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

beadon@semiauto:~$ dkms status
8812au/5.6.4.2_35491.20191025, 6.2.0-23-generic, x86_64: installed
8812au/5.6.4.2_35491.20191025, 6.2.0-24-generic, x86_64: installed
nvidia/535.54.03, 6.2.0-24-generic, x86_64: installed
virtualbox/7.0.6, 6.2.0-23-generic, x86_64: installed
virtualbox/7.0.6, 6.2.0-24-generic, x86_64: installed
beadon@semiauto:~$ dpkg-reconfigure nvidia-dkms
/usr/sbin/dpkg-reconfigure must be run as root
beadon@semiauto:~$ sudo dpkg-reconfigure nvidia-dkms-535
Removing all DKMS Modules
Done.
update-initramfs: deferring update (trigger activated)
INFO:Enable nvidia
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/lenovo_thinkpad
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/put_your_quirks_here
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/dell_latitude
Loading new nvidia-535.54.03 DKMS files...
Building for 6.2.0-24-generic
Building for architecture x86_64
Building initial module for 6.2.0-24-generic
Done.

nvidia.ko:
Running module version sanity check.
Module version 535.54.03 for nvidia.ko
exactly matches what is already found in kernel 6.2.0-24-generic.
DKMS will not replace this module.
You may override by specifying --force.

nvidia-modeset.ko:
Running module version sanity check.
Module version 535.54.03 for nvidia-modeset.ko
exactly matches what is already found in kernel 6.2.0-24-generic.
DKMS will not replace this module.
You may override by specifying --force.

nvidia-drm.ko:
Running module version sanity check.
Module version 535.54.03 for nvidia-drm.ko
exactly matches what is already found in kernel 6.2.0-24-generic.
DKMS will not replace this module.
You may override by specifying --force.

nvidia-uvm.ko:
Running module version sanity check.
Module version 535.54.03 for nvidia-uvm.ko
exactly matches what is already found in kernel 6.2.0-24-generic.
DKMS will not replace this module.
You may override by specifying --force.

nvidia-peermem.ko:
Running module version sanity check.
Module version 535.54.03 for nvidia-peermem.ko
exactly matches what is already found in kernel 6.2.0-24-generic.
DKMS will not replace this module.
You may override by specifying --force.
depmod...
Processing triggers for initramfs-tools (0.142ubuntu2) ...
update-initramfs: Generating /boot/initrd.img-6.2.0-24-generic

hmm, modules are in place ..

[beadon (bryant-eadon)]

setting the PRIME profile back into place for NVIDIA.

This causes the gdm issues, I suspect because this promotes the eGPU to primary and there is some kind of resource sharing problem.

So, the way it MUST be configured is "ON DEMAND" PRIME -- which allows the eGPU to be detected, and allows logins to occur with Xorg. However, I am almost sure that this means most rendering is done via the Intel GPU, and not the NVIDIA GPU.

I can see this lack of utilization by using nvidia-settings, I see the GPU utilization is 0%.

It now has me curious if there is an intel-gpu-equivalent utilization tool?

[beadon (bryant-eadon)]

Notably, when I "enable 3d" with a windows 10 guest on VirtualBox, I can see the GPU being actively used!

Now, if there was only a way to use it with Xorg on the Host machine, not just the guest.

[beadon (bryant-eadon)]

This may be unrelated -- but I am also seeing "x86/split lock detection" errors. Is this just a red herring ? Only seeing these when using the VirtualBox to drive the GPU.

[Tue Jul 11 14:50:44 2023] usb 2-2.2: current rate 16000 is different from the runtime rate 48000
[Tue Jul 11 14:50:44 2023] usb 2-2.2: current rate 16000 is different from the runtime rate 48000
[Tue Jul 11 14:50:44 2023] usb 2-2.2: current rate 16000 is different from the runtime rate 48000
[Tue Jul 11 14:50:45 2023] Bluetooth: RFCOMM TTY layer initialized
[Tue Jul 11 14:50:45 2023] Bluetooth: RFCOMM socket layer initialized
[Tue Jul 11 14:50:45 2023] Bluetooth: RFCOMM ver 1.11
[Tue Jul 11 14:50:47 2023] thunderbolt 1-1: new device found, vendor=0x127 device=0x2
[Tue Jul 11 14:50:47 2023] thunderbolt 1-1: Razer Core X Chroma
[Tue Jul 11 14:50:48 2023] rfkill: input handler disabled
[Tue Jul 11 14:50:53 2023] thunderbolt 1-101: new device found, vendor=0x127 device=0x3
[Tue Jul 11 14:50:53 2023] thunderbolt 1-101: Razer Core X Chroma
[Tue Jul 11 14:50:54 2023] rfkill: input handler enabled
[Tue Jul 11 14:50:54 2023] usb 2-2.2: current rate 16000 is different from the runtime rate 48000
[Tue Jul 11 14:50:54 2023] usb 2-2.2: current rate 16000 is different from the runtime rate 48000
[Tue Jul 11 14:50:54 2023] usb 2-2.2: current rate 16000 is different from the runtime rate 48000
[Tue Jul 11 14:50:59 2023] rfkill: input handler disabled
[Tue Jul 11 14:56:33 2023] SUPR0GipMap: fGetGipCpu=0x1b
[Tue Jul 11 14:56:33 2023] vboxdrv: 0000000000000000 VMMR0.r0
[Tue Jul 11 14:56:33 2023] vboxdrv: 0000000000000000 VBoxDDR0.r0
[Tue Jul 11 14:56:41 2023] x86/split lock detection: #AC: EMT-1/4206 took a split_lock trap at address: 0x7f698f296453
[Tue Jul 11 14:57:03 2023] x86/split lock detection: #AC: EMT-2/4207 took a split_lock trap at address: 0x7f698f296453
[Tue Jul 11 14:57:03 2023] x86/split lock detection: #AC: EMT-0/4205 took a split_lock trap at address: 0x7f698f296453

Unsure why there is a magic address space number there: 0x7f698f296453

[beadon (bryant-eadon)]

Appears to be a red herring, VirtualBox apparently experiences random guest crashes when split lock detection is enabled on the Host OS. ( I am able to verify this )

Reference:
"Random crashes with Windows 10 guest operating system with Intel Tiger Lake chipset"

CPU is :
https://ark.intel.com/content/www/us/en/ark/products/208664/intel-core-i71185g7-processor-12m-cache-up-to-4-80-ghz-with-ipu.html

11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
(yes, it's Tiger Lake).

The reference in the VirtualBox ticketing system seems to have been deleted, it appears that VirtualBox is using split locking.

Reference to the much longer discussion of what this is and how it's impacting systems is here :
https://lwn.net/Articles/911219/

seems there's a penalty applied when the code is doing split locking. This is a tunable.

Deeper discussion of how Intel and the Linux kernel team were involved is here:
https://www.virtualbox.org/ticket/20180?cversion=0&cnum_hist=4

Here are the errors again for the split lock, note the gnome-shell errors just before :

2023-07-11T16:01:26.427798-07:00 semiauto gnome-character[5478]: JS LOG: Characters Application exiting
2023-07-11T16:01:29.507946-07:00 semiauto gnome-shell[2700]: Can't update stage views actor <unnamed>[<MetaWindowActorX11>:0x55c018ffa2a0] is on because it needs an allocation.
2023-07-11T16:01:29.508086-07:00 semiauto gnome-shell[2700]: Can't update stage views actor <unnamed>[<MetaSurfaceActorX11>:0x55c01daf3ce0] is on because it needs an allocation.
2023-07-11T16:01:29.601935-07:00 semiauto gnome-shell[2700]: Can't update stage views actor <unnamed>[<MetaWindowActorX11>:0x55c01ab19670] is on because it needs an allocation.
2023-07-11T16:01:29.602058-07:00 semiauto gnome-shell[2700]: Can't update stage views actor <unnamed>[<MetaSurfaceActorX11>:0x55c018212df0] is on because it needs an allocation.
2023-07-11T16:01:37.587046-07:00 semiauto kernel: [ 2385.543465] x86/split lock detection: #AC: EMT-0/5871 took a split_lock trap at address: 0x7fd78aa96453
2023-07-11T16:01:38.079070-07:00 semiauto kernel: [ 2386.033592] x86/split lock detection: #AC: EMT-1/5872 took a split_lock trap at address: 0x7fd78aa96453
2023-07-11T16:01:39.723080-07:00 semiauto kernel: [ 2387.676986] x86/split lock detection: #AC: EMT-2/5873 took a split_lock trap at address: 0x7fd78aa96453
2023-07-11T16:01:45.484475-07:00 semiauto systemd[1]: systemd-timedated.service: Deactivated successfully.
2023-07-11T16:02:22.331078-07:00 semiauto kernel: [ 2430.285901] pcieport 0000:00:07.2: pciehp: Slot(0-1): Link Down
2023-07-11T16:02:22.331102-07:00 semiauto kernel: [ 2430.285908] pcieport 0000:00:07.2: pciehp: Slot(0-1): Card not present

The split lock penalty triggered by VirtualBox seems to trip some kind of timer which causes the link to the eGPU to be reset. Which then of course causes all kinds of havoc on the system.

Further logs ;
2023-07-11T16:07:55.895042-07:00 semiauto kernel: [ 2763.846900] thunderbolt 0000:00:0d.2: failed to send driver ready to ICM

So -- long way around , there are a lot of things interacting in here. I am unsure how to get a pure Xorg work-around in place to use the extra GPU. I am also a litt...

[Daniel van Vugt (vanvugt)]

This bug is closed because it's a duplicate of bug 1970291.

If you have found any new bugs in comments #8 - #18 then please open new bug reports for them.

[beadon (bryant-eadon)]

Sure,I get it.

The work around of using Xorg does not resolve the issue on this setup, Wayland and Xorg both suffer different problems. Just wanted to call that out!

Simple open questions:

1. The NVIDIA DRM component here appears to be a culprit. Can I disable it ? Is there an open source driver that performs better than NVIDIA's proprietary driver here in this configuration you have run across ?

2. Exercising the NVIDIA GPU by using a VM (VirtualBox) was used as a method to troubleshoot and use it in a real world scenario. It may have run afoul of other bugs. Is there a better way to exercise the NVIDIA GPU and/or the Intel GPU and stay within the context of this ticket ?

3. gdm failing(?) to start when the NVIDIA GPU is set as PRIME, preventing GUI login seems like a bug. Would you like an Xorg(?) bug ticket for this ?

4. Please be advised, selecting Xorg as a server from the gdm login screen was never an option for me. Should I be concerned that a version or code difference here is leading us to different results ?

5. split_lock detection has knock-on effects due to the delay introduced, this was only highlighted by kernel logs using the the VirtualBox implementation when I began to exercise the GPU. Can we safely rule out split_lock's for this ticket ? Or is it worth troubleshooting further in the event I am experiencing the effects, but they're simply not getting reliably *detected* by the Tiger Lake or split_lock kernel logic ?

Thanks.

[Daniel van Vugt (vanvugt)]

I think all of those questions are off-topic for this bug so should not be discussed here. That said...

1. No that's not relevant. It's required to put a picture on the screen.

2. I don't know.

3. Please log a new bug for that unless you're referring to https://bugs.launchpad.net/ubuntu/+source/nvidia-graphics-drivers-535/+bug/2026789/comments/7 in which case it's a feature of the nvidia driver. Not a bug.

4. Please log a new bug for that.

5. That's a general kernel question I don't have the answers to. Please log a new bug or question to the 'linux' source package for such general kernel issues.