AMDGPU VMC page fault with Athlon 200GE APU

[Steven Ellis]

System
 - AMD Athlon 200GE
 - Gigabyte GA-B450M-S2H board - latest F2c Firmware
 - Ubuntu 18.04.1 LTS

Reproducible with kernels
 - linux-image-generic-hwe-18.04 version 4.18.0.13.63
 - linux-image-unsigned-4.20.0-042000rc4-generic_4.20.0-042000rc4.201812030528_amd64.deb
---
ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
DistroRelease: Ubuntu 18.04
InstallationDate: Installed on 2016-08-15 (872 days ago)
InstallationMedia: Mythbuntu 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.1)
Package: linux (not installed)
Tags: bionic
Uname: Linux 4.20.0-042000rc4-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: Upgraded to bionic on 2019-01-03 (1 days ago)
UserGroups:

_MarkForUpload: True
---
ProblemType: Bug
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
AudioDevicesInUse:
 USER PID ACCESS COMMAND
 /dev/snd/controlC0: mythfe 2175 F.... mythfrontend.re
                      mythfe 2181 F.... pulseaudio
 /dev/snd/pcmC0D3p: mythfe 2175 F...m mythfrontend.re
DistroRelease: Ubuntu 18.04
HibernationDevice: RESUME=/dev/mythnew/swap
InstallationDate: Installed on 2016-08-15 (872 days ago)
InstallationMedia: Mythbuntu 16.04 LTS "Xenial Xerus" - Release amd64 (20160420.1)
MachineType: Gigabyte Technology Co., Ltd. B450M S2H
Package: linux (not installed)
ProcFB: 0 amdgpudrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.18.0-13-generic root=/dev/mapper/mythnew-myth_root ro verbose
ProcVersionSignature: Ubuntu 4.18.0-13.14~18.04.1-generic 4.18.17
PulseList: Error: command ['pacmd', 'list'] failed with exit code 1: No PulseAudio daemon running, or not running as session daemon.
RelatedPackageVersions:
 linux-restricted-modules-4.18.0-13-generic N/A
 linux-backports-modules-4.18.0-13-generic N/A
 linux-firmware 1.173.2
RfKill:

Tags: bionic
Uname: Linux 4.18.0-13-generic x86_64
UpgradeStatus: Upgraded to bionic on 2019-01-03 (1 days ago)
UserGroups:

_MarkForUpload: True
dmi.bios.date: 12/04/2018
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: F2c
dmi.board.asset.tag: Default string
dmi.board.name: B450M S2H
dmi.board.vendor: Gigabyte Technology Co., Ltd.
dmi.board.version: x.x
dmi.chassis.asset.tag: Default string
dmi.chassis.type: 3
dmi.chassis.vendor: Default string
dmi.chassis.version: Default string
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvrF2c:bd12/04/2018:svnGigabyteTechnologyCo.,Ltd.:pnB450MS2H:pvrDefaultstring:rvnGigabyteTechnologyCo.,Ltd.:rnB450MS2H:rvrx.x:cvnDefaultstring:ct3:cvrDefaultstring:
dmi.product.family: Default string
dmi.product.name: B450M S2H
dmi.product.sku: Default string
dmi.product.version: Default string
dmi.sys.vendor: Gigabyte Technology Co., Ltd.

[muelladdi (muelladdi-linux-kernel-bugs)]

System video freezes randomly during the day while working on the system normally (no 3D).
Logfile shows only:

May 17 11:18:00 ws01 kernel: [11831.268044] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring sdma0 timeout, last signaled seq=2081253, last emitted seq=2081256
May 17 11:18:00 ws01 kernel: [11831.268051] [drm] No hardware hang detected. Did some blocks stall?

If I can assist any further, please tell.

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

Can you load kernel with grub command line amdgpu.vm_update_mode=3 to force CPU VM update mode and see if this makes the issue go away ?

Andrey

[muelladdi (muelladdi-linux-kernel-bugs)]

Added the grub cmdline and will investigate

[song.fc (song.fc-linux-kernel-bugs)]

Hello, I think I'm experiencing the same problem here. My Ryzen 5 2400G system freezes often, especially under some high cpu and disk activity, even after the "Typical Current Idle" UEFI workaround. Sometimes I could reboot with sysrq, most of the times I need a hard reset. The freezes never leave a message in kernel log until today, after 2 months I built my machine.

6月 16 23:29:00 sfc-DESKTOP kernel: [drm] No hardware hang detected. Did some blocks stall?
6月 16 23:29:00 sfc-DESKTOP kernel: [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, last signaled seq=1217227, last emitted seq=1217229
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: at page 0x000000011241a000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: at page 0x0000000112414000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: at page 0x0000000112416000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: at page 0x0000000112410000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: at page 0x0000000112412000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: at page 0x000000011241f000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: at page 0x0000000112425000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: at page 0x0000000112427000 from 27
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
6月 16 23:28:50 sfc-DESKTOP kernel: amdgpu 0000:38:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
6月 16 23:28:50 sfc-DESKTOP...

[chewi (chewi-linux-kernel-bugs)]

Not sure if this is related to bug #199653, which concerns freezing on the 2500U and 2700U. It hasn't received any attention from AMD or other kernels devs but there is more information that could potentially be useful. I tried amdgpu.vm_update_mode=3 but that didn't help.

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

Those two bugs are unrelated.

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

(In reply to James Le Cuirot from comment #4)
> Not sure if this is related to bug #199653, which concerns freezing on the
> 2500U and 2700U. It hasn't received any attention from AMD or other kernels
> devs but there is more information that could potentially be useful. I tried
> amdgpu.vm_update_mode=3 but that didn't help.

Are you seeing sdma0 timeout message when the system freezes like muelladi above ? I expect
amdgpu.vm_update_mode=3 to help only in that case.

[chewi (chewi-linux-kernel-bugs)]

(In reply to Andrey Grodzovsky from comment #6)
> Are you seeing sdma0 timeout message when the system freezes like muelladi
> above ? I expect
> amdgpu.vm_update_mode=3 to help only in that case.

I haven't been able to get any information as I have been unable to access the system following these freezes. Judging by the output from the other reporter, they do indeed seem unrelated. Sorry for the noise but some attention on that issue would be hugely appreciated. It's so bad, I've considered selling the laptop.

[song.fc (song.fc-linux-kernel-bugs)]

Today I tried the amdgpu.vm_update_mode=3 option, and my computer still freezed. This time no log was recorded.

[song.fc (song.fc-linux-kernel-bugs)]

I found I'm able to reproduce the freeze when I'm compiling two Android ROMs at the same time while browsing or something else. It will freeze in at most 3 hours. I have a 300W PSU, and 2 SSD + 1 HDD storages. Android sources are stored in the HDD. This time I can reboot with sysrq, seems not a PSU fault.

[muelladdi (muelladdi-linux-kernel-bugs)]

So far I have not had any more freezes during normal, non-3D work.

[michel (michel-linux-kernel-bugs)]

Other Raven Ridge users have reported that updating to the current microcode files from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amdgpu has fixed stability issues.

Make sure your system BIOS and CPU microcode are up to date as well.

[song.fc (song.fc-linux-kernel-bugs)]

I updated to the current microcode files at linux-firmware.git. Today I replicated the same workload, and my computer freezed in just 5 minutes! Even sysrq does not work. I have to do a hard reset.

Still, no log was recorded.

I have an Asrock AB350M-Pro4 motherboard, with latest UEFI version L4.82.

[michel (michel-linux-kernel-bugs)]

(In reply to notsyncing from comment #12)
> I updated to the current microcode files at linux-firmware.git.

Did you update the microcode files in the initrd as well?

[song.fc (song.fc-linux-kernel-bugs)]

(In reply to Michel Dänzer from comment #13)
> Did you update the microcode files in the initrd as well?

I just copied the files to /lib/firmware. I'll try with update-initramfs.

[song.fc (song.fc-linux-kernel-bugs)]

Still freezed under two Android source compilation + 2 intellij idea + 10 firefox tabs + EVE online playing after 3 hours. Sysrq does not work, need hard reset. No log was recorded.

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

What kernel version are you using ?

We can try and figure out what was the last commands in HW before you experienced the page fault.

You can clone and install our register analyzer from here - https://cgit.freedesktop.org/amd/umr/

Then launch your X with ENV variable GALLIUM_DDEBUG=always to dump all the 3D commands into files in ~/ddebug_dumps/

Run your workload.

After you again experience the GPU page fault please provide the following outputs

sudo umr -lb
sudo umr -O verbose,follow_ib -R gfx[.]
sudo umr -O bits -wa
sudo umr -O many,bits -r*.*.mmGRBM_STATUS
sudo umr -O many,bits -r *.*.HEADER_DUMP
sudo umr -O many,bits -r *.*.CP_EOP

[song.fc (song.fc-linux-kernel-bugs)]

(In reply to Andrey Grodzovsky from comment #16)
> Then launch your X with ENV variable GALLIUM_DDEBUG=always to dump all the
> 3D commands into files in ~/ddebug_dumps/

Would you mind telling me how to add this variable? I googled and cannot find any information. Should I add it to /etc/X11/Xsession or something else? Thanks very much!

I'm on kernel 4.17.2-041702-generic, Ubuntu 18.04, mesa 18.1.1-1ubuntu1~18.04.0~ppa1

[song.fc (song.fc-linux-kernel-bugs)]

BTW, I'm using KDE plasma 5.13 with sddm.

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

(In reply to notsyncing from comment #18)
> BTW, I'm using KDE plasma 5.13 with sddm.

You just prepend this before command to start your graphic stack. You need to manually run your graphic stack from command line and add this before. E.G. for me on Ubuntu I will disable graphics launch on boot from GRUB. Then from terminal I will run
GALLIUM_DDEBUG=always service lightdm start

[song.fc (song.fc-linux-kernel-bugs)]

(In reply to Andrey Grodzovsky from comment #20)
> You just prepend this before command to start your graphic stack. You need
> to manually run your graphic stack from command line and add this before.
> E.G. for me on Ubuntu I will disable graphics launch on boot from GRUB. Then
> from terminal I will run
> GALLIUM_DDEBUG=always service lightdm start

I compiled and installed umr and executed "GALLIUM_DDEBUG=always service sddm start" on a root tty, and no "ddebug_dumps" directory was found in /root or in my home directory after desktop started. Is that normal?

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

(In reply to notsyncing from comment #21)
> (In reply to Andrey Grodzovsky from comment #20)
> > You just prepend this before command to start your graphic stack. You need
> > to manually run your graphic stack from command line and add this before.
> > E.G. for me on Ubuntu I will disable graphics launch on boot from GRUB.
> Then
> > from terminal I will run
> > GALLIUM_DDEBUG=always service lightdm start
>
> I compiled and installed umr and executed "GALLIUM_DDEBUG=always service
> sddm start" on a root tty, and no "ddebug_dumps" directory was found in
> /root or in my home directory after desktop started. Is that normal?

I checked myself, you need to use this variable with specific graphic application you are running, can you pinpoint what graphic work is going on while you have this faults ?

[song.fc (song.fc-linux-kernel-bugs)]

In fact there is no specified graphic work. I just put an Android source compilation running and go to bed, besides only firefox and dolphin running. The next morning, I found the machine has already freezed. Still no log. The log I posted is just a lucky one.

Would you mind telling me if the umr had some system-wide debugging methods? Or should I make a netconsole to see if there was anything?

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

(In reply to notsyncing from comment #23)
> In fact there is no specified graphic work. I just put an Android source
> compilation running and go to bed, besides only firefox and dolphin running.
> The next morning, I found the machine has already freezed. Still no log. The
> log I posted is just a lucky one.
>
> Would you mind telling me if the umr had some system-wide debugging methods?
> Or should I make a netconsole to see if there was anything?

UMR is system wide any way, it's memory/registers/HW debugging tool. you can provide the UMR outputs I asked before once the freeze happened assuming you still have SSH access (which seems like you don't).

Since the memory faults you experience are clearly due to some graphic rendering activity maybe you could try to isolate the app which triggers it, repeat what you do but close both firefox and dolphin and check if this still happens. If not try to find which of them was causing this and then we can run it with MESA debug flags.

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

Created attachment 277059
Trace process

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

Created attachment 277061
Trace process 2

Attached 2 patches if applied to your kernel should tell which process caused the VM_FAULT please also launch your kernel from GRUB with following parameter -
amdgpu.vm_fault_stop=2

[song.fc (song.fc-linux-kernel-bugs)]

After 3 days, I managed to reproduce it again with 2 android compilation and firefox with kernel parameter mem=4096m (I have 16GB memory). I found that it's easier to reproduce when the memory is full.

sfc@sfc-DESKTOP:~$ sudo umr -lb
        raven1.gfx91
        raven1.vcn10
        raven1.dcn10
        raven1.nbio70
        raven1.sdma041
        raven1.hdp40
        raven1.oss40
        raven1.mmhub91
        raven1.mp100
sfc@sfc-DESKTOP:~$ sudo umr -O verbose,follow_ib -R gfx[.]
error: Unknown option [follow_ib]
sfc@sfc-DESKTOP:~$ sudo umr -O bits -wa
No active waves!
sfc@sfc-DESKTOP:~$ sudo umr -O many,bits -r*.*.mmGRBM_STATUS
[ERROR]: Unknown option <-r*.*.mmGRBM_STATUS>
sfc@sfc-DESKTOP:~$ sudo umr -O many,bits -r *.*.HEADER_DUMP
gfx91.mmCP_MEC_ME1_HEADER_DUMP => 0xc0000e00
        .HEADER_DUMP[0:31] == 3221229056 (0xc0000e00)
gfx91.mmCP_MEC_ME2_HEADER_DUMP => 0xdef0def0
        .HEADER_DUMP[0:31] == 3740327664 (0xdef0def0)
gfx91.mmCP_ME_HEADER_DUMP => 0xc0004200
        .ME_HEADER_DUMP[0:31] == 3221242368 (0xc0004200)
gfx91.mmCP_PFP_HEADER_DUMP => 0xffff1000
        .PFP_HEADER_DUMP[0:31] == 4294905856 (0xffff1000)
gfx91.mmCP_CE_HEADER_DUMP => 0xffff1000
        .CE_HEADER_DUMP[0:31] == 4294905856 (0xffff1000)
sfc@sfc-DESKTOP:~$ sudo umr -O many,bits -r *.*.CP_EOP
gfx91.mmCP_EOPQ_WAIT_TIME => 0x0000052c
        .WAIT_TIME[0:9] == 300 (0x0000012c)
        .SCALE_COUNT[10:17] == 1 (0x00000001)
gfx91.mmCP_EOP_DONE_ADDR_LO => 0x00609000
        .ADDR_LO[2:31] == 1582080 (0x00182400)
gfx91.mmCP_EOP_DONE_ADDR_HI => 0x000000f5
        .ADDR_HI[0:15] == 245 (0x000000f5)
gfx91.mmCP_EOP_DONE_DATA_LO => 0x000008e4
        .DATA_LO[0:31] == 2276 (0x000008e4)
gfx91.mmCP_EOP_DONE_DATA_HI => 0x00000000
        .DATA_HI[0:31] == 0 (0x00000000)
gfx91.mmCP_EOP_LAST_FENCE_LO => 0x000008e4
        .LAST_FENCE_LO[0:31] == 2276 (0x000008e4)
gfx91.mmCP_EOP_LAST_FENCE_HI => 0x00000000
        .LAST_FENCE_HI[0:31] == 0 (0x00000000)
gfx91.mmCP_EOP_DONE_EVENT_CNTL => 0x00038060
        .WBINV_TC_OP[0:6] == 96 (0x00000060)
        .WBINV_ACTION_ENA[12:17] == 56 (0x00000038)
        .CACHE_POLICY[25:25] == 0 (0x00000000)
        .EXECUTE[28:28] == 0 (0x00000000)
gfx91.mmCP_EOP_DONE_DATA_CNTL => 0x40010000
        .DST_SEL[16:17] == 1 (0x00000001)
        .INT_SEL[24:26] ...

[song.fc (song.fc-linux-kernel-bugs)]

btw, these umr commands are executed after reboot.

[song.fc (song.fc-linux-kernel-bugs)]

I just upgraded to mesa 18.1.3 and kernel 4.17.3. I ran firefox with GALLIUM_DDEBUG after reboot. It produces these after I opened some tabs and firefox stopped responsing.

---
Gallium debugger active. Logging all calls.
Hang detection timeout is 1000ms.
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000000
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000001
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000002
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000003
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000004
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000005
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000006
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000007
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000008
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000009
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000010
dd: can't create a directory (13)
dd: can't create a directory (13)
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000011
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000012
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000013
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000014
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000015
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000016
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000017
dd: can't create a directory (13)
dd: failed to open /home/sfc/ddebug_dumps/firefox_5057_00000018
GPU hang detected, collecting information...

Draw # driver prev BOP TOP BOP dump file
-------------------------------------------------------------
8 YES NO NO NO dd: can't create a directory (13)
fopen failed

Done.
Sandbox: seccomp sandbox violation: pid 5057, tid 5268, syscall 162, args 140631883336544 7 140633020008624 140632174061559 7 140632344285188.
dd: Aborting the process...
[Parent 4748, Gecko_IOThread] WARNING: pipe error (72): 连接被对方重设: file /build/firefox-m9FtQy/firefox-60.0.2+build1/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353
[Parent 4748, Gecko_IOThread] WARNING: pipe error (113): 连接被对方重设: file /build/firefox-m9FtQy/firefox-60.0.2+build1/ipc/chromium/src/chrome/common/ipc_channel_posix.cc, line 353

###!!! [Parent][MessageChannel] Error: (msgtype=0x15007F,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv

---

The umr commands are:

---
sfc@sfc-DESKTOP:~$ sudo umr -lb
        raven1.gfx91
        raven1.vcn10
        raven1.dcn10
        raven1.nbio70
        raven1.sdma041
        raven1.hdp40...

[song.fc (song.fc-linux-kernel-bugs)]

Created attachment 277101
Kernel log from serial port when it freezes

Finally got logs from serial port when freezed. Seems my problem has nothing to do with amdgpu. Maybe I should file a new bug.

[chewi (chewi-linux-kernel-bugs)]

(In reply to notsyncing from comment #30)
> Finally got logs from serial port when freezed. Seems my problem has nothing
> to do with amdgpu. Maybe I should file a new bug.

I may be off the mark but that looks more like bug #196683. Have you tried adjusting "Power Supply Idle Control" in the BIOS (if you have it) or using zenstates.py to disable the C6 package state?

[song.fc (song.fc-linux-kernel-bugs)]

I've set that option to "Typical Current Idle" and still freezes. The logs in 196683 points to RCU, which seems not my case. I suspect it's due to the zram. Now I'm trying to reproduce it with zram disabled.

[song.fc (song.fc-linux-kernel-bugs)]

Now about half a month passed, and my machine is running fine with zram disabled. No more freezes. Seems my problem is related to zram, not this bug, while the "ring gfx timeout" error never happened again. Thanks for everyone who gave me advice.

[andrey.grodzovsky (andrey.grodzovsky-linux-kernel-bugs)]

notsyncing, can you close this ticket then ?

[song.fc (song.fc-linux-kernel-bugs)]

(In reply to Andrey Grodzovsky from comment #34)
> notsyncing, can you close this ticket then ?

This is not my ticket.

[Steven Ellis (steven-openmedia)]

Example error message with 4.18.0.13.63 kernel

[11569.691337] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691344] amdgpu 0000:0a:00.0: at page 0x0000000106200000 from 27
[11569.691348] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00101031
[11569.691356] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691359] amdgpu 0000:0a:00.0: at page 0x0000000106200000 from 27
[11569.691362] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[11569.691370] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691373] amdgpu 0000:0a:00.0: at page 0x0000000106200000 from 27
[11569.691376] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[11569.691383] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691386] amdgpu 0000:0a:00.0: at page 0x0000000106201000 from 27
[11569.691389] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[11569.691396] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691399] amdgpu 0000:0a:00.0: at page 0x0000000106200000 from 27
[11569.691402] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[11569.691409] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691412] amdgpu 0000:0a:00.0: at page 0x0000000106201000 from 27
[11569.691415] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[11569.691422] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691425] amdgpu 0000:0a:00.0: at page 0x0000000106200000 from 27
[11569.691428] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[11569.691435] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691438] amdgpu 0000:0a:00.0: at page 0x0000000106201000 from 27
[11569.691441] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[11569.691448] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691451] amdgpu 0000:0a:00.0: at page 0x0000000106201000 from 27
[11569.691454] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[11569.691461] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768)
[11569.691464] amdgpu 0000:0a:00.0: at page 0x0000000106204000 from 27
[11569.691467] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[11579.766498] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring gfx timeout, last signaled seq=42143, last emitted seq=42144
[11579.766504] [drm] GPU recovery disabled.

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

Reproduced with Mainline kernel

wget -c https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.20-rc4/linux-image-unsigned-4.20.0-042000rc4-generic_4.20.0-042000rc4.201812030528_amd64.deb https://kernel.ubuntu.com/~kernel-ppa/mainline/v4.20-rc4/linux-modules-4.20.0-042000rc4-generic_4.20.0-042000rc4.201812030528_amd64.deb

apt-get install linux*4.20*

New error message

[ 6891.737234] gmc_v9_0_process_interrupt: 14 callbacks suppressed
[ 6891.737240] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768, for process Xorg pid 1741 thread amdgpu_cs:0 pid 1793)
[ 6891.737247] amdgpu 0000:0a:00.0: in page starting at address 0x0000000101300000 from 27
[ 6891.737250] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00101031
[ 6891.737259] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768, for process Xorg pid 1741 thread amdgpu_cs:0 pid 1793)
[ 6891.737262] amdgpu 0000:0a:00.0: in page starting at address 0x0000000101300000 from 27
[ 6891.737265] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[ 6891.737274] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768, for process Xorg pid 1741 thread amdgpu_cs:0 pid 1793)
[ 6891.737277] amdgpu 0000:0a:00.0: in page starting at address 0x0000000101300000 from 27
[ 6891.737279] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[ 6891.737287] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768, for process Xorg pid 1741 thread amdgpu_cs:0 pid 1793)
[ 6891.737290] amdgpu 0000:0a:00.0: in page starting at address 0x0000000101301000 from 27
[ 6891.737293] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[ 6891.737301] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768, for process Xorg pid 1741 thread amdgpu_cs:0 pid 1793)
[ 6891.737303] amdgpu 0000:0a:00.0: in page starting at address 0x0000000101300000 from 27
[ 6891.737306] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[ 6891.737314] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768, for process Xorg pid 1741 thread amdgpu_cs:0 pid 1793)
[ 6891.737317] amdgpu 0000:0a:00.0: in page starting at address 0x0000000101301000 from 27
[ 6891.737319] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[ 6891.737327] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768, for process Xorg pid 1741 thread amdgpu_cs:0 pid 1793)
[ 6891.737330] amdgpu 0000:0a:00.0: in page starting at address 0x0000000101300000 from 27
[ 6891.737332] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[ 6891.737340] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768, for process Xorg pid 1741 thread amdgpu_cs:0 pid 1793)
[ 6891.737343] amdgpu 0000:0a:00.0: in page starting at address 0x0000000101301000 from 27
[ 6891.737345] amdgpu 0000:0a:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
[ 6891.737353] amdgpu 0000:0a:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:1 pasid:32768, for process Xorg pid 1741 thread amdgpu_cs:0 pid 1793)
[ 6891.737356] amdgpu 0000...

[Steven Ellis (steven-openmedia)]

This appears to be similar to

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1767667
​https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1804505
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1772081

[Steven Ellis (steven-openmedia)]

Note that crash is happening with MPEG2 and H.264 video playback under MythTV

[Steven Ellis (steven-openmedia)]

Potential upstream kernel issue

https://bugzilla.kernel.org/show_bug.cgi?id=199749

[Steven Ellis (steven-openmedia)]

# lspci -nn | grep -E 'VGA|Display'
0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega [Radeon Vega 8 Mobile] [1002:15dd] (rev cb)

[Steven Ellis (steven-openmedia)]

Another potential upstream BZ
 - https://bugs.freedesktop.org/show_bug.cgi?id=108098

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

apport information

[Steven Ellis (steven-openmedia)]

Switched back to the Stock 18.04 LTS kernel and installed
 - amdgpu-pro-18.50-708488-ubuntu-18.04
 - https://www.amd.com/en/support/kb/release-notes/rn-rad-lin-18-50-unified

With straight amdgpu support VDPAU doesn't work correctly and I needed to install the full amdgpu-pro version. So far system is stable and I haven't had any new amdgpu lockups.

I'd prefer to use the pure amdgpu opensource drive, but even the upstream 4.20 kernel has issues.

[experimancer (experimancer)]

I have it too: (random hangups of whole system either under load/idle)
- Ryzen5 2400 G
- Ubuntu 18.04 and 18.10
- kernel 4.18

Log: /var/log/syslog (or kern.log) juts before the freeze, after which only har reset, power off/on makes ysstem boot agian:

Jan 15 02:34:13 elrond kernel: [ 476.185425] gmc_v9_0_process_interrupt: 32 callbacks suppressed
Jan 15 02:34:13 elrond kernel: [ 476.185429] amdgpu 0000:0b:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 v
mid:3 pasid:32768)
Jan 15 02:34:13 elrond kernel: [ 476.185433] amdgpu 0000:0b:00.0: at page 0x000000010780a000 from 27
Jan 15 02:34:13 elrond kernel: [ 476.185434] amdgpu 0000:0b:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00301031
Jan 15 02:34:13 elrond kernel: [ 476.185440] amdgpu 0000:0b:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 v
mid:3 pasid:32768)
Jan 15 02:34:13 elrond kernel: [ 476.185441] amdgpu 0000:0b:00.0: at page 0x000000010780b000 from 27
Jan 15 02:34:13 elrond kernel: [ 476.185443] amdgpu 0000:0b:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
Jan 15 02:34:13 elrond kernel: [ 476.185448] amdgpu 0000:0b:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 v
mid:3 pasid:32768)
Jan 15 02:34:13 elrond kernel: [ 476.185449] amdgpu 0000:0b:00.0: at page 0x0000000107805000 from 27
Jan 15 02:34:13 elrond kernel: [ 476.185450] amdgpu 0000:0b:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
Jan 15 02:34:13 elrond kernel: [ 476.185455] amdgpu 0000:0b:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:3 pasid:32768)
Jan 15 02:34:13 elrond kernel: [ 476.185457] amdgpu 0000:0b:00.0: at page 0x0000000107806000 from 27
Jan 15 02:34:13 elrond kernel: [ 476.185458] amdgpu 0000:0b:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
Jan 15 02:34:13 elrond kernel: [ 476.185463] amdgpu 0000:0b:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:3 pasid:32768)
Jan 15 02:34:13 elrond kernel: [ 476.185464] amdgpu 0000:0b:00.0: at page 0x0000000107808000 from 27
Jan 15 02:34:13 elrond kernel: [ 476.185465] amdgpu 0000:0b:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
Jan 15 02:34:13 elrond kernel: [ 476.185470] amdgpu 0000:0b:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:3 pasid:32768)
Jan 15 02:34:13 elrond kernel: [ 476.185471] amdgpu 0000:0b:00.0: at page 0x0000000107809000 from 27
Jan 15 02:34:13 elrond kernel: [ 476.185473] amdgpu 0000:0b:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
Jan 15 02:34:13 elrond kernel: [ 476.185478] amdgpu 0000:0b:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:3 pasid:32768)
Jan 15 02:34:13 elrond kernel: [ 476.185479] amdgpu 0000:0b:00.0: at page 0x0000000107803000 from 27
Jan 15 02:34:13 elrond kernel: [ 476.185480] amdgpu 0000:0b:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
Jan 15 02:34:13 elrond kernel: [ 476.185485] amdgpu 0000:0b:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:3 pasid:32768)
Jan 15 02:34:13 elrond kernel: [ 476.185486] amdgpu 0000:0b:00.0: at page 0x0000000107804000 from 27
Jan 15 02:34:13 elrond kernel: [ 476.185487] amdgpu 0000:0b:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00000000
Jan 15 02:34:13 elrond kernel: [ 476.185492] amdgpu 0000:0b:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:3 pa...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Luca (zapduke)]

This fixes it for me (raven ridge 2200g though). But unfortunately it seems that the version in linux-firmware is not sufficient and after every update it gets back.

[Paul Crawford (psc-sat)]

I am seeing the same sort of bug on my machine.
msi X470 motherboard (just tried latest firmware - no change)
AMD Ryzen 5 2400G and Ubuntu 18.04
Linux paul-ubuntu 4.18.0-25-generic #26~18.04.1-Ubuntu SMP Thu Jun 27 07:28:31 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Sometimes it happens overnight when machine is idle / screen locked, other times when using machine. Oddly enough the most recent one still had the mouse moving but keyboard unresponsive. Also a remote reboot fails - unless I use the watchdog with 'nowayout=1' to for a hard reset 60s after things should have gone down so it seem systemd won't/can't stop the hung graphics system.
Typically the error messages look like:

[ 2567.463753] amdgpu 0000:27:00.0: [gfxhub] VMC page fault (src_id:0 ring:24 vmid:3 pasid:32770)
[ 2567.463763] amdgpu 0000:27:00.0: at page 0x000000010be01000 from 27
[ 2567.463767] amdgpu 0000:27:00.0: VM_L2_PROTECTION_FAULT_STATUS:0x00301031

Often repeated many, many times.
Seeing this bug report this has been going on for over a year now it seems and still is not fixed! Also a friend of mine is seeing the same on another AMD-based system. All pretty crap and makes me seriously regret buying an AMD board.

[John Rose (johnaaronrose)]

I'm getting same on new AMD Ryzen 3270U with Ubuntu 22.04.