Ryzen 1800X freeze - rcu_sched detected stalls on CPUs/tasks

Bug #1690085 reported by Vincent on 2017-05-11
208
This bug affects 35 people
Affects Status Importance Assigned to Milestone
Linux
Confirmed
Medium
linux (Ubuntu)
High
Unassigned

Bug Description

Hi,

We aregetting various kernel crash on a pretty new config.
We're using Ryzen 1800X CPU with X370 Gaming Pro Carbon MB (7A32V1) using latest BIOS available (1.52)

We are running Ubuntu 17.04 (amd64), we've tried different kernel version, native one and releases from http://kernel.ubuntu.com/~kernel-ppa/mainline/ too.
Tested kernel version:

native 17.04 kernel
4.10.15

Issues are the same, we're getting random freeze on the machine.

Here is kern.log entry when happening :

May 10 22:41:56 dev2 kernel: [24366.186246] INFO: rcu_sched detected stalls on CPUs/tasks:
May 10 22:41:56 dev2 kernel: [24366.187618] 0-...: (1 GPs behind) idle=49b/1/0 softirq=28561/28563 fqs=913449
May 10 22:41:56 dev2 kernel: [24366.188977] (detected by 12, t=1860207 jiffies, g=10001, c=10000, q=4656)
May 10 22:41:56 dev2 kernel: [24366.190344] Task dump for CPU 0:
May 10 22:41:56 dev2 kernel: [24366.190345] swapper/0 R running task 0 0 0 0x00000008
May 10 22:41:56 dev2 kernel: [24366.190348] Call Trace:
May 10 22:41:56 dev2 kernel: [24366.190354] ? native_safe_halt+0x6/0x10
May 10 22:41:56 dev2 kernel: [24366.190355] ? default_idle+0x20/0xd0
May 10 22:41:56 dev2 kernel: [24366.190358] ? arch_cpu_idle+0xf/0x20
May 10 22:41:56 dev2 kernel: [24366.190360] ? default_idle_call+0x23/0x30
May 10 22:41:56 dev2 kernel: [24366.190362] ? do_idle+0x16f/0x200
May 10 22:41:56 dev2 kernel: [24366.190364] ? cpu_startup_entry+0x71/0x80
May 10 22:41:56 dev2 kernel: [24366.190366] ? rest_init+0x77/0x80
May 10 22:41:56 dev2 kernel: [24366.190368] ? start_kernel+0x464/0x485
May 10 22:41:56 dev2 kernel: [24366.190369] ? early_idt_handler_array+0x120/0x120
May 10 22:41:56 dev2 kernel: [24366.190371] ? x86_64_start_reservations+0x24/0x26
May 10 22:41:56 dev2 kernel: [24366.190372] ? x86_64_start_kernel+0x14d/0x170
May 10 22:41:56 dev2 kernel: [24366.190373] ? start_cpu+0x14/0x14
May 10 22:44:56 dev2 kernel: [24546.188093] INFO: rcu_sched detected stalls on CPUs/tasks:
May 10 22:44:56 dev2 kernel: [24546.189461] 0-...: (1 GPs behind) idle=49b/1/0 softirq=28561/28563 fqs=935027
May 10 22:44:56 dev2 kernel: [24546.190823] (detected by 14, t=1905212 jiffies, g=10001, c=10000, q=4740)
May 10 22:44:56 dev2 kernel: [24546.192191] Task dump for CPU 0:
May 10 22:44:56 dev2 kernel: [24546.192192] swapper/0 R running task 0 0 0 0x00000008
May 10 22:44:56 dev2 kernel: [24546.192195] Call Trace:
May 10 22:44:56 dev2 kernel: [24546.192199] ? native_safe_halt+0x6/0x10
May 10 22:44:56 dev2 kernel: [24546.192201] ? default_idle+0x20/0xd0
May 10 22:44:56 dev2 kernel: [24546.192203] ? arch_cpu_idle+0xf/0x20
May 10 22:44:56 dev2 kernel: [24546.192204] ? default_idle_call+0x23/0x30
May 10 22:44:56 dev2 kernel: [24546.192206] ? do_idle+0x16f/0x200
May 10 22:44:56 dev2 kernel: [24546.192208] ? cpu_startup_entry+0x71/0x80
May 10 22:44:56 dev2 kernel: [24546.192210] ? rest_init+0x77/0x80
May 10 22:44:56 dev2 kernel: [24546.192211] ? start_kernel+0x464/0x485
May 10 22:44:56 dev2 kernel: [24546.192213] ? early_idt_handler_array+0x120/0x120
May 10 22:44:56 dev2 kernel: [24546.192214] ? x86_64_start_reservations+0x24/0x26
May 10 22:44:56 dev2 kernel: [24546.192215] ? x86_64_start_kernel+0x14d/0x170
May 10 22:44:56 dev2 kernel: [24546.192217] ? start_cpu+0x14/0x14

Depending on the kernel version, we've got NMI watchdog errors related to CPU stuck (mentioning the CPU core id, which is random).
Crash is happening randomly, but in general after some hours (3-4h).

Now, we've installed kernel 4.11.0-041100-generic #201705041534 this morning and waiting for crash...
For now, the machine is not "used", at least, it's not CPU stressed...

Thanks
---
ApportVersion: 2.20.4-0ubuntu4
Architecture: amd64
DistroRelease: Ubuntu 17.04
InstallationDate: Installed on 2017-05-09 (1 days ago)
InstallationMedia: Ubuntu-Server 17.04 "Zesty Zapus" - Release amd64 (20170412)
Package: linux (not installed)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=fr_FR.UTF-8
 SHELL=/bin/bash
Tags: zesty
Uname: Linux 4.11.0-041100-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True

Vincent (hvincent13) wrote :

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1690085

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete

apport information

tags: added: apport-collected zesty
description: updated

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Lipido (lipido) wrote :

Hi Vicent,

Did you experience more crashes with kernel 4.11?

Thank you!

Vincent (hvincent13) wrote :

Hi Lipido,

Yes, i'm still experiencing crashes, even when using 4.11

Please find full kern.log in attachment.

Regards,

Lipido (lipido) wrote :

Ok, me too :-(

Crashes appear from 24-48h of operation.

My hardware is:
- SSUS PRIME B350-Plus
- Amd Ryzen 5 1600

OS: Ubuntu 16.04

Tried:
- Update BIOS to the latest (v 609)
- Kernel 4.10
- Disable SMT in bios (from 12 threads to 6 threads)
- Boot clocksource=tsc iommu=soft
- Disable IOMMU in bios

None of these work. I was waiting for 4.11 to be available for Ubuntu as an official package, but it seems that this will not work either.

Another link talking about what seems to be the same issue:

https://forum.level1techs.com/t/ryzen-vs-ubuntu/115715/22

Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.12 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12-rc1/

Changed in linux (Ubuntu):
importance: Undecided → High
status: Confirmed → Incomplete
tags: added: kernel-da-key
Vincent (hvincent13) wrote :

Hi Joseph,

I've just installed 4.12-rc1.
Now waiting for a crash... or not (hope so !)

Regards,

Vincent (hvincent13) wrote :

Just go a kernel panic...

How to add the tag kernel-bug-exists-upstream ?

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
tags: added: kernel-bug-exists-upstream
Joseph Salisbury (jsalisbury) wrote :

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

Please follow the instructions on the wiki page[0]. The first step is to email the appropriate mailing list. If no response is received, then a bug may be opened on bugzilla.kernel.org.

Once this bug is reported upstream, please add the tag: 'kernel-bug-reported-upstream'.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

Changed in linux (Ubuntu):
status: Confirmed → Triaged
Vincent (hvincent13) wrote :

Hi,

Could you please tell me which address I have to send the mail to?
I don't really understand how to achieve this bug report on the mailing list.

Thanks,

Vincent (hvincent13) wrote :

Hi,

It looks like that the system is stable when removing "nouveau" driver.
Waiting 24/48h and will post again.

Regards,

camparijet (iichikolamp) wrote :

Hi Vincent,

I also met the problem, and your advice removing "nouveau" succeeded for work-around to stabilize system. Before it freeze every 3-5hr after booting up, but now working without problem for 2 days.

My enviroment is:
- cpu: Ryzen 1700
- motherboard: ASUS B350M-A
- graphics card: NVIDIA GK208
- kernel: 4.11.0-041100rc8-generic
- dist: 17.04

Vincent (hvincent13) wrote :

Hi,

5 days update now and no crash.

camparijet => did you installed NVIDIA driver instead ?

Regards,

camparijet (iichikolamp) wrote :

Hi Vincent,

> camparijet => did you installed NVIDIA driver instead ?

No. I don't have to use the card for my purpose, so simply i disable it.

Alex Jones (blenheimears) wrote :

I'm seeing this crash even with the Nvidia official driver.

Alex Jones (blenheimears) wrote :

This is a hardware bug in the CPU. This ticket should be closed as invalid.

Changed in linux (Ubuntu):
status: Triaged → Invalid
Alex Jones (blenheimears) wrote :

Reopening because even though this is a known issue with the CPU we could still implement a workaround. One workaround is to disable address space layout randomization:

echo 0 >/proc/sys/kernel/randomize_va_space

However, that would be disabling a security feature.

Changed in linux (Ubuntu):
status: Invalid → New

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
tags: added: artful
Marc Rene Schädler (suaefar) wrote :

Was this issue officially confirmed to be a hardware bug in Ryzen processors by AMD?
If so, could you provide a link to the statement?

Disabling address space layout randomization (ASLR) seems to alleviate the problem, but does it solve it?

I am investigating unstable behavior under load which could be related.
There, disabling ASLR is not sufficient!
People suggest to increase SOC voltages, use specific versions of the kernel and the like.
See https://community.amd.com/thread/215773?start=135&tstart=0 for more info.

Alex Jones (blenheimears) wrote :

AMD has not publicly commented on this issue that I'm aware of. This issue has been seen on many different operating systems. DragonFlyBSD includes a workaround for this issue. The workaround on Linux is to compile the kernel with CONFIG_RCU_NOCB_CPU, CONFIG_RCU_NOCB_CPU_ALL, and disable ASLR using "echo 0 >/proc/sys/kernel/randomize_va_space". This can be put into rc.local. It's also possible to hardcode ASLR as disabled into the kernel, but this requires modifying the kernel source, not just the config file. There is a new AGESA update released about a week ago, 1.0.0.6a, although I have not tested whether the new AGESA alone (without any kernel changes) solves the issue.

Alex Jones (blenheimears) wrote :

I just tested 1.0.0.6a AGESA, and it does not solve this issue. In some cases just one program will crash, and in other cases the entire system will crash. I will test the workaround above later.

Alex Jones (blenheimears) wrote :

If any of your RAM timings are odd (eg. 17), setting them to the next even number (eg. 18) helps a lot. Recompiling the kernel with CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_ALL, and disabling ASLR is still necessary though. It may also be a good idea to give the SOC slightly more voltage, but not more than 1.2 V.

Kai-Heng Feng (kaihengfeng) wrote :

Alex,

Can you provide link on how DragonflyBSD fixed this issue?

Alex Jones (blenheimears) wrote :

This is the DragonFlyBSD commit. https://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/b48dd28447fc8ef62fbc963accd301557fd9ac20

It appears that there are two different ways that the system can crash, which is why it is necessary to both disable ASLR and to compile the kernel with CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_ALL. The ASLR-related crash usually results in a single or a few programs crashing (although if an important program crashes it can bring down the entire system) and happens under heavy load. The other crash only happens if the kernel was compiled without CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_ALL (Ubuntu's kernel is compiled without these options) and happens when the system is idle or nearly idle, and results in a complete system crash.

Alex Jones (blenheimears) wrote :

If the motherboard allows it, disabling the OpCache will completely prevent (or at least greatly reduce the probability of) the ASLR-related crash, even if ASLR is enabled in the kernel. As far as I'm aware it has no effect on the other type of crash.

Alex Jones (blenheimears) wrote :

I've also determined that changing the CPU, memory, or SOC voltages or timings has little or no effect on either type of crash.

Kai-Heng Feng (kaihengfeng) wrote :

This is beyond my expertise - let's see what upstream can do.

Torge (cyslider) wrote :

I also experience this problem since I updated yesterday.

I am using Kubuntu 16.04 with KDE backports enabled.
I also have a Ryzen 1800X and an Asrock X370 Gaming professional

I experience this either when I boot and don't log in promptly or when I enter the lockscreen.

After reading this page I tried to run
  seq 1 | xargs -P0 -n1 md5sum /dev/zero &

as root to always keep one CPU core busy and indeed this seems to prevent the lockscreen problem...

I would also like to note that the system was not 100% frozen, once every few minutes it seemed to response for a short time, enabling me to switch to the TTY. Then I always saw some processes hanging at 100% that I could not kill, or the kill was delayed for a long time. I always see the errors in the initial posts during that time. The TTY seemes to work fine though, once entered though.

David Wilson (plottt) wrote :

I've ran into this problem several times. After disabling C-states in the motherboard, my system has been running stable for ~5 weeks uptime so far.

Torge (cyslider) wrote :

Ok, this trick did not help and my PC did not make it through the night without freezing again.

However I finally found the real problem. I installed oibaf a while back as my Kubuntu was flickering all over the place. Now it seems the be the source of my problem. I deinstalled oibaf again and now everything seems fine. No lock screen freezing for me anymore.

Stuart Page (sdpagent) wrote :

Just wanting to report that I am also experiencing this issue with a freshly installed Ubuntu Server 16.04.3 with 4.10 kernel on the 22nd August 2017. All the updates applied and running as a KVM host with hardware:

Ryzen 1700 (non-x)
Motherboard: Asus prime B350-Plus
Bios: - Version 0805
 - Disabled SMT
 - Disabled c-states

xb5i7o (xb5i7o) wrote :

Hi guys,

Im getting the same issues, brand new build.

Ryzen 1800x
Asrock X370 Taichi
BIOS: 3.10 (latest)
I tried disabling cool n quiet and c-state

However there is another option under advanced - for GLOBAL C-States that i just disabled today and i am waiting to see what will happen.

On BIOS v3.00 PC would freeze after 6 hours of idling or not touching anything. Come back to see my keyboard and mouse and everything was frozen.
With BIOS v.3.10 now i get random reboots atleast once a day.

Im running Ubuntu 16.04.3 Kernel 4.4.0-93

Anyone know to direct me where the SoC voltage would be in BIOS? is it the VDD SoC?
Anyone know how to update my kernel to 4.10 if it tells me i have the latest?

Stuart Page (sdpagent) wrote :

I finally managed to figure out how to compile a kernel with RCU_NOCB and disabled ASLR as Alex Jones mentioned, and it appears to have worked for me and another guy who helped me put the tutorial together:

http://blog.programster.org/ubuntu-16-04-compile-custom-kernel-for-ryzen

Kai-Heng Feng (kaihengfeng) wrote :

Stuart,

Does this issue also happen on latest mainline kernel?

Franck Charras (franckc) wrote :

Hi,
I'm getting the same issues on several identical builds, with an ASUS prime X370-PRO motherboard.
It's very hard to analyze since it happens randomly every other week, and it leaves no logs. It seems that the freeze happens at idle after very high memory load (observation after logging CPU and RAM loads).
Compiling the kernel as suggested in this thread didn't work (new freeze this morning).

tgui (eric-c-morgan) wrote :

I too still have random computer shutdowns without logs. Uptime varies from a couple days to a week or so. It happens it seems after being relatively idle for a long period. I do have high memory usage because of the VMs I run.

I've disabled C-states, cool and quiet, tested memory, and so forth. I also compiled a new kernel as mentioned by Stuart. I do not have segfaults with compilations.

I am willing to run tests and provide info. Please let me know if anyone wants something.

Ubuntu 16.04
Asrock x370 ITX
32GB Ram
Ryzen 1700

Franck Charras (franckc) wrote :

Ryzen CPUs manufactured before week 24 of this year were known to have issues (especially the segfault issue). It was officially fixed for all ryzen manufactured after week 30. All my ryzen are pre-24 CPUs and they all have this silent crash issue. Does it also happen with the post week 30 ryzen ? (@tgui what is yours ?)

AS (as2008) wrote :

I can confirm the issue with a week 33 Ryzen 1700 on Asus Prime B350 Plus.
I had segfaults with my previous 1700 (week 22 iirc). No more segfaults with the week 33 CPU but still random crashes on (long time) idle.

tags: removed: kernel-da-key
information type: Public → Public Security
information type: Public Security → Public
Changed in linux:
importance: Unknown → Medium
status: Unknown → Confirmed
429 comments hidden view all 509 comments

Adding to the confusion here:

I'm running a Ryzen 1700 on Asus PRIME B350-PLUS motherboard and on BIOS version 4011 I have the "Power Supply Idle Control" option and I have changed it from "Auto" to "Low Current Idle" a week or so ago and after that I have had zero lockups.

According to this bug report disabling C6 states *or* setting that option to "Typical Current Idle" should work and "Auto" should mean "Low Current Idle" hence it should not have made any difference. zenstates.py shows C6 enabled for both package and core.

Unless the Ubuntu 18.04 kernel 4.15.0-*-generic got an off-tree patch applied somewhere in-between my reboots I'm confused and relieved at the same time.

Did the ppl with the new BIOSes double test the 'Power Supply Idle Control' option by disabling it temporary? Because, it could also be fixed by that Agesa update some of you mentioned (also mentioned in [1])

tl;dr:
Had soft lockup at full load, BIOS-update seems to help, but it doesn't have the 'Power Supply Idle Control'-option.

More details:
My HP 17-ca0202ng (Ryzen 2500U) was stable by starting up with "idle=nomwait" Kernel-parameter for 2~3 weeks until *something*[2] happened and from one day to the next it was hard-locking again and on another testrun it threw the soft-lock-error (NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s!) while it was transcoding x264-stuff to x265 (kind of full load; 95% @ all 8 threads). It was bootet up by "nomodeset" at the time the errors occur, so I'm quite sure it can't be amdgpu-related.
The time the soft-lock occurred, the system was kind of responsive, just freezing after a amount of seconds with unfreezing after another amount of seconds. I was able to run dmesg and just saw that NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! for all threads, before more freezings in shorter amount of seconds appeared and the system completely froze. Strange!

But: HP provided a BIOS-Update (F4 > F10) some days ago, I was able to update it yesterday and I've done 8 hours of full-load testing, 8 hours of idle testing and 7,5 hours of partial-load testing. No lockup so far and no Kernel-parameters needed (running Ubuntu's mainline build kernel 4.17.2, the stuff from Padoka stable ppa and ravenridge firmware from [3] to prevent amdgpu-freezes). As any lockup occurred within 3 hours, the laptop seems to be stable (or at least "more stable" than before).
Can't say anything about the changes HP made in that update, because they don't provide a changelog. But it's a laptop, so it has 3 or 4 BIOS-options (something virtualisation something, something secure boot something and some boot order options) and for sure no 'Power Supply Idle Control'. Allow me to point out, that Laptops usually have power adapters special designed for the device. So it can't be power supply related as AMD said (IMO).

[1] https://community.amd.com/thread/225795 , https://community.amd.com/thread/224000
[2] May be microcode-update related provided by Ubuntu, but I can't remember for sure.
[3] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/amdgpu

Interestingly,

I had a nicely running system, uptime about 30 days, and I just decided to upgrade the Kernel to the latest (Fedora 28), and "what the heck" I will just upgrade the motherboard BIOS too.

Mobo: ASUSTeK model: PRIME B350M-A
6 core AMD Ryzen 5 1600X Six-Core (-MT-MCP-) arch: Zen rev.1

So, now I have BIOS version 4104 (previous was 3801 I think)

And after like 10 minutes of doing stuff (i.e. not idle), LOCKED UP !!!

So I went into the BIOS and changed the Global C-State to Disabled, and right under that is the power profile, changed it to Typical.

We shall see how long it lasts !

I guess "don't fix what ain't broken" applied to my activity today lol.

(In reply to Robert Hoffmann from comment #386)
> Interestingly,
>
> I had a nicely running system, uptime about 30 days, and I just decided to
> upgrade the Kernel to the latest (Fedora 28), and "what the heck" I will
> just upgrade the motherboard BIOS too.
>
> Mobo: ASUSTeK model: PRIME B350M-A
> 6 core AMD Ryzen 5 1600X Six-Core (-MT-MCP-) arch: Zen rev.1
>
> So, now I have BIOS version 4104 (previous was 3801 I think)
>
> And after like 10 minutes of doing stuff (i.e. not idle), LOCKED UP !!!
>
> So I went into the BIOS and changed the Global C-State to Disabled, and
> right under that is the power profile, changed it to Typical.
>
> We shall see how long it lasts !
>
> I guess "don't fix what ain't broken" applied to my activity today lol.

My case is similar, but i don't recall updating the BIOS/UEFI firmware, it was after a kernel upgrade that the system started locking up :/

I have the ASUS B350M board and I updated to latest BIOS 4014 but my lock ups started with the latest kernel 4.17 in Arch where the lock up bug reappeared. I have set the idle current setting to typical and so far no lock up in two days, lets see what happens in few days. Keeping my fingers crossed.

I'm watching this thread since February. My first stable workaround with legacy BIOS-Mode was the kernel parameter rcu_nocbs=0-15 because the BIOS didn't have the "Typical Current Idle" Option.

Systeminfo
R7 1700, GA AX370-Gaming K7 with BIOS F23f in legacy mode, PSU Corsair HX750, OS Gentoo Linux Kernel 4.17.4, desktop kde-plasma-5.13.2

no overclocking or tweaking, only using the "Typical Current Idle" Option (it's possible now with F23f) instead "Auto".

So I didn't need rcu_nocbs=0-15 furthermore and all works fine - no lockups

These lockups are probably not related to this bug. I've updated by Intel Sandy Bridge laptop to 4.17.5 from Fedora 28 repository and now I have random CPU lockups, too.
4.17.3 worked fine.

(In reply to ValdikSS from comment #391)
> These lockups are probably not related to this bug. I've updated by Intel
> Sandy Bridge laptop to 4.17.5 from Fedora 28 repository and now I have
> random CPU lockups, too.
> 4.17.3 worked fine.

Please take a look at this report, it matches your description https://bugzilla.redhat.com/show_bug.cgi?id=1598989 The user that reported it also has an Intel CPU (Acer Aspire V3-771 from his attached screenshot). I have experienced this issue once on one of my Ryzen systems running 4.17.6, could it be an unrelated problem? Another one https://bugzilla.redhat.com/show_bug.cgi?id=1598462

With regards to the original Ryzen Random Soft Lockup issue: I am witnessing it on every kernel that I have tested so far, ranging from kernel 4.10 through to 4.17.6. The frequency of the Ryzen soft lockup has increased since kernel >= 4.15 in my experience. It could just be a random effect as I am unaware of the cause of this problem.

I am running my machines at stock clocks and have the latest stable BIOS updates install as of today on a Ryzen 1700 w/ MSI PRIME X370-PRO, Ryzen 1800X w/ ASRock Fatal1ty X370 Professional Gaming, and finally my personal desktop Ryzen 1800X w/ X470 Taichi Ultimate w/ Seasonic 1000W Platinum PSU (Haswell ready). All of my CPUs are running microcode patch level 0x8001137

I can't find any correlation in workload, the issue occurs on web servers, VFIO gaming, even live USB sessions. I used the following (try) to provide insight.
journalctl -t kernel --no-pager | grep "soft lockup" | awk -F"!" '{print $2}' | sort -u
 [Compositor:4167]
 [kworker/10:1:18249]
 [kworker/1:3:418]
 [libvirtd:1226]
 [systemd:1]
 [Web Content:3505]

I have attempted CPU pinning, disabling ASLR completely. I have also tried isolating workloads in virtual machines that do not cross CCX units with hugepages (THP off). I am not an expert in knowledge of CCX, NUMA, Infinity Fabric, etc... I could have made a mistake in my test. That said, I am currently testing the "idle=nomwait" parameter in hope of getting better results, who knows perhaps even a stable system.

There still does not appear to be a single consistent solution to this problem and it exists in Ryzen 1xxx and 2xxx series.

Apparently not in the Epyc series, is that correct?

This thread is utterly embarrassing for AMD, truly embarrassing.

(In reply to OptionalRealName from comment #393)
> There still does not appear to be a single consistent solution to this
> problem and it exists in Ryzen 1xxx and 2xxx series.

Actually, there are several workarounds known to work listed in this thread.

I've been running a stable x1800 using the rcu_nocbs=0-15 fixes noted above for almost a year now.

> This thread is utterly embarrassing for AMD, truly embarrassing.

It's a nuanced hard to troubleshoot bug in a processor + support chipsets with a known workaround which only seems to happen on Linux... not really sure how that is "embarrassing". It sucks, it's new silicon. Welcome to being an early adopter.

Look at Intel's heart-bleed and all the other major security catastrophes they've had in recent history. Stuff happens. I do wish AMD would address it, but they're likely reluctant to bring it up until they have a reliable fix (if it even is a fix on their end)

> It's a nuanced hard to troubleshoot bug in a processor + support chipsets
> with a known workaround which only seems to happen on Linux... not really
> sure how that is "embarrassing". It sucks, it's new silicon. Welcome to
> being an early adopter.

Thread Reported: 2017-08-16 19:05 UTC

https://www.anandtech.com/show/11170/the-amd-zen-and-ryzen-7-review-a-deep-dive-on-1800x-1700x-and-1700
16 month old silicon.

Nah this is too long.
Firstly, nothing should need to be done in linux in any capacity, there should be a bios flag that fixes it.
Second, the bios flag should be on at all times
Third, based on my second point, as long as someone has the latest firmware, it should 100% solve the problem.

Users should not need to mess around for stability, that is not acceptable.

I have Ryzen 7 1800X on Asus Prime X370-Pro. I upgraded the BIOS to v4011(Update AGESA 1.0.0.2a + SMU 43.18) and:

  1) turned on the "Typical Current Idle" option.

  2) stopped using zenstates.py -- which I had been using to enable "C6 Core"
     but disable "C6 Package" (to no avail).

  3) did *not* change Linux -- which was 4.16.5 -- Fedora 27.

  4) continued to use CONFIG_RCU_NOCB_CPU and rcu_nocbs=0-15

After 67 days uptime (leaving the system completely idle and changing nothing), I became convinced that the "Typical Current Idle" option has dealt with the "freezing when idle" problem.

When I say "freezing when idle", what I mean is: if the machine is left idle (typically over night) it simply stops responding. Nothing at all is logged -- no application, driver or kernel errors or warnings are logged -- the machine is still powered up, but frozen solid. The only way to restart the machine is to power down and up again.

Reviewing this thread, it seems to be mostly concerned with the "freezing while idle" issue.

The symptoms of the original "Random Soft Lockup" include log messages of the form:

     NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s!

is that related to "freezing while idle", or is it a separate issue ?

I get the impression that CONFIG_RCU_NOCB_CPU and rcu_nocbs=0-15 may be related to the "Random Soft Lockup"... but not to "freezing while idle" ???

It seems that other crashes/lockups are trying to attach themselves to this thread.

I note that this bug is asigned to <email address hidden>. This bug is very nearly 1 year old. Is this a good moment for the assignee to address this thread and say:

  * what, if any, Kernel issues have been identified

  * what, if any, Kernel fixes have been applied

related to this thread.

If the root cause of (some or all of) the issues in this thread is fixed or worked around by the "Typical Current Idle" BIOS option, does the assignee think that this "bug" can now be closed, or are there actual Kernel issues that remain, waiting to be fixed ?

Is it significant that W*nd*rs does not seem to suffer ?

This bug should be closed as "FIXED" in my humble opinion. "NEW" is grossly inaccurate.

My workstation uptime is now 109 days, continuous operation without suspend, mainly idle, mixed with episodes of heavy load on all cores and everything in between. The technical term for this is "spectacularly stable". Credit belongs to all involved, including AMD, kernel devs, motherboard maker, Debian maintainers, and many other groups who made this possible. Again in my humble opinion, the only issue remaining is AMD's failure to explain what went wrong and exactly how they worked around it. That is a separate bug, this one is done.

I suspect that Windows users did get this bug, but they didn't notice it because, well use your imagination. Then it was quietly fixed in a Windows update supplied by AMD that went out around the same time as the Typical Power bios updates.

Antoine Pitrou (pitrou) wrote :

This bug still occurs for me with 4.15.0-32-generic (Ubuntu 18.04.1) if I don't disable C6 states.

@Daniel what makes you think this bug is fixed? It certainly isn't for me. I have a ASRock Fatal1ty X370 Gaming X motherboard, a Ryzen 7 1700 CPU, have updated the motherboard BIOS to its latest version (4.80 with PinnaclePI-AM4_1.0.0.2 Patch A) and am still getting idle lockups with kernel 4.15.0-32-generic (Ubuntu 18.04.1 kernel). The workaround for me is to disable C6 states using the Zenstates.py script (which has the unfortunate side effect of lowering the max CPU frequency in single-threaded mode).

(I also don't see any "Typical Current Idle" option in my Setup options, though I might have overlooked it)

Hello there. I read most of the thread a week ago.

My machine was freezing when idle and when not idle. I read most of this thread and thanks to it got a stable setup.

@Daniel: When you say that you think the problem is fixed, do you know what change/Linux version fixes the issue for most people?

-------------

Now my report.

I have the following setup:

Motherboard: AB350M-DS3H (F23d BIOS).
CPU: Ryzen 2700X.
No overclock.

I still wonder whether my CPU is doing too much extra work (I don't know what cpufreq would report if a C6 state was reached).

cpufreq stats:

CPU 0: 3.70 GHz:6.64%, 3.20 GHz:1.15%, 2.20 GHz:92.21%
CPU15: 3.70 GHz:4.86%, 3.20 GHz:0.80%, 2.20 GHz:94.34%

Anyway, to make the things work I had to use three tweaks. I tried individual tweaks to no avail.

- Select in BIOS: "Typical Current Idle"
- Start Linux with: idle=nomwait
- Disable C6 states (both core and package) with Zenstates.py

Before doing the last step Zenstates.py reports Core enabled and Package disabled.

I'm using Ubuntu 18.04.01 LTS. I didn't compile Linux with CONFIG_RCU_NOCB_CPU / CONFIG_RCU_NOCB_CPU_ALL.

So, things are working for me. But if you think I should test a new Linux version that is supposed to fix the issue please let me know.

(In reply to Antoine Pitrou from comment #398)
> @Daniel what makes you think this bug is fixed? It certainly isn't for me.
> I have a ASRock Fatal1ty X370 Gaming X motherboard, a Ryzen 7 1700 CPU, have
> updated the motherboard BIOS to its latest version (4.80 with
> PinnaclePI-AM4_1.0.0.2 Patch A) and am still getting idle lockups with
> kernel 4.15.0-32-generic (Ubuntu 18.04.1 kernel). The workaround for me is
> to disable C6 states using the Zenstates.py script (which has the
> unfortunate side effect of lowering the max CPU frequency in single-threaded
> mode).
>
> (I also don't see any "Typical Current Idle" option in my Setup options,
> though I might have overlooked it)

Hi Antoine, i checked the manual of your Asrock motherboard quickly.
For your machine the "typical idle current" is named "c6 mode"
Disable it and you get the "typical idle current", i think.
While keeping the boost monothread.

The "freezing while idle" problem does appear to be (at least) worked around by the "Typical Current Idle" option. If that counts as a fix, then the bug is fixed -- or rather, was never a Kernel Bug.

The "Random Soft Lockup" -- the nominal subject of the bug -- on the other hand... who can tell ?

It would be churlish not to be happy that my machine is now reliable and I am a believer in credit where credit is due.

So, as you say, all credit goes to AMD for their total silence up to and after the release of the work around (or fix) -- except for the advice that some (older) power supplies might not work. I don't really need to know why or how the new option does the trick, but I remain curious whether it has a material effect on power consumption.

But I also give credit to this bug's assignee for their total silence -- in particular:

  * it's not clear whether the "freezing while idle" and "Random Soft Lockup"
    problems are separate, related or the same.

  * hence, the CONFIG_RCU_NOCB_CPU and rcu_nocbs=0-15 voodoo may or may
    not still be recommended ?

    For all I know, with the "Typical Current Idle" option, I may be better
    off returning to the default configuration ?

  * and there is talk of kernel changes which reduce power consumption
    (significantly ?) -- which may or may not be related to these issues.

Sadly, the absence of solid information allows half-truths and ill-informed speculation to take its place and to live on in the undamped echo chamber of the interweb :-( [I acknowledge my small part in that noise :-(]

(In reply to Daniel Phillips from comment #397)
> This bug should be closed as "FIXED" in my humble opinion. "NEW" is grossly
> inaccurate.
>
> My workstation uptime is now 109 days, continuous operation without suspend,
> mainly idle, mixed with episodes of heavy load on all cores and everything
> in between. The technical term for this is "spectacularly stable". Credit
> belongs to all involved, including AMD, kernel devs, motherboard maker,
> Debian maintainers, and many other groups who made this possible. Again in
> my humble opinion, the only issue remaining is AMD's failure to explain what
> went wrong and exactly how they worked around it. That is a separate bug,
> this one is done.
>
> I suspect that Windows users did get this bug, but they didn't notice it
> because, well use your imagination. Then it was quietly fixed in a Windows
> update supplied by AMD that went out around the same time as the Typical
> Power bios updates.

It seems than the normal power mode under this OS disable core parking (C6)
I say it because i checked myself and core parking doesn't work by default under w10.
So it could explain why it's not a windows thing.

To anyone applying the BIOS/CONFIG_RCU_NOCB_CPU/ZenStates solution together with the idle=nomwait solution, I believe this is unnecessary as these are different problems relating to different CPUs. The former worked for my Ryzen 5 1600X, while only the latter worked for my much newer Ryzen 7 2700U (Raven Ridge). I've heard the idle=nomwait issue applies to all Raven Ridge CPUs. I don't know if that's true but I'm fairly certain it doesn't apply to any earlier hardware.

The title of the bug should be changed to include Ryzen 7 1700X, and the reporter should follow up, if the problem still occurs with Linux 4.18.

For the other issues, separate bug reports should be submitted. Though the mailing lists might be the better forum for this, as the Linux kernel developers for the subsystem do not seem to use the Kernel.org Bugzilla.

(In reply to Antoine Pitrou from comment #398)
> @Daniel what makes you think this bug is fixed? It certainly isn't for me.
> I have a ASRock Fatal1ty X370 Gaming X motherboard, a Ryzen 7 1700 CPU, have
> updated the motherboard BIOS to its latest version (4.80 with
> PinnaclePI-AM4_1.0.0.2 Patch A) and am still getting idle lockups with
> kernel 4.15.0-32-generic (Ubuntu 18.04.1 kernel). The workaround for me is
> to disable C6 states using the Zenstates.py script (which has the
> unfortunate side effect of lowering the max CPU frequency in single-threaded
> mode).

Please remember, this is the upstream Linux kernel bug tracker. So, reports should only be for the latest stable releases, which currently is 4.18.3 [1]. Some distributions make it easy to install those. Packages are available for Ubuntu [2].

Please report issues with the heavily patched Ubuntu default Linux kernel to the Ubuntu bug tracker [3].

[1]: https://www.kernel.org/
[2]: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.18.3/
[3]: https://bugs.launchpad.net/ubuntu/+source/linux/+bugs

@Paul I understand this. I'm also monitoring the corresponding Ubuntu issue. Still, I was asking whether there was a specific reason to believe this bug was fixed upstream (as opposed to the fact that a single person doesn't experience the issue anymore).

(In reply to Michaël Colignon from comment #400)
>
> Hi Antoine, i checked the manual of your Asrock motherboard quickly.
> For your machine the "typical idle current" is named "c6 mode"
> Disable it and you get the "typical idle current", i think.
> While keeping the boost monothread.

Hi Michaël, that option was removed in the latest BIOS updates. Probably because it didn't work at all.

(In reply to Antoine Pitrou from comment #407)
> (In reply to Michaël Colignon from comment #400)
> >
> > Hi Antoine, i checked the manual of your Asrock motherboard quickly.
> > For your machine the "typical idle current" is named "c6 mode"
> > Disable it and you get the "typical idle current", i think.
> > While keeping the boost monothread.
>
> Hi Michaël, that option was removed in the latest BIOS updates. Probably
> because it didn't work at all.

Oh yes it works. On others boards it fix the crashes in idle.
Contact asrock support about the "typical idle current" needed option.

I don't think there is any reason to think that the "c6 mode" option had anything to do with "typical idle current". Its name implied that it disabled C6 states, which it actually didn't (according to the Zenstates script).

I have an Asus ROG Strix GL702ZC notebook with Ryzen 7 1700 + Radeon RX580, and I have soft lockups too. The computer completly freeze when idle. My system is an Ubuntu 18.04.1. Kernel parameters like "idle=nomwait" and "rcu_nocbs=0-15" don't fix my problem. Now, I'm using my computer "without lockups" with cpu governor setted in "performance". My BIOS, updated to 305, does not have the option "Typical Idle Current" or similar.

Unfortunately, setting the governor to performance doesn't fix the issue here (Ryzen 7 1700, ASRock Fatal1ty X370 Gaming X).

It appears that selecting the "Typical Current Idle" BIOS option (where available) eliminates the "freeze when idle" problem (somehow).

That said, it also appears that this is *not* the default.

The need for the option appears to be Linux specific, or at least it is not needed for W*nd*rs.

That said, I am damned if I can find any documentation for the option, or any authoritative recommendation for its use with Linux.

So, anyone bitten by this problem may struggle to find their way to selecting "Typical Current Idle". Particularly as the symptoms have all the appearance of a hardware issue -- possibly an overclocking step too far.

Would it be possible for the Kernel to set whatever the "Typical Current Idle" BIOS option sets ? Or invoke some BIOS function to select the option ? That way, at least an up to date version of Linux would avoid this problem "out of the box".

I have been clapping my hands vigorously for some time, but I am coming to the conclusion that the <email address hidden> fairy does not, in fact, exist (or perhaps not in this universe).

While I am practising believing the regulation six impossible things before breakfast, I note that the <email address hidden> fairy would have to be able to communicate with the AMD documentation fairy... perhaps using a 56k ouija board ?

Finally and FWIW, I tried turning off "rcu_nocbs=0-15" and "Typical Current Idle", and my Ryzen 7 1800X/ASUS X370-Pro/Linux 4.18/Fedora froze within hours. Before the "Typical Current Idle" option became available, I was running "rcu_nocbs=0-15" (following the advice in this thread), and my machine froze every week or so. I have now been running for some days with "Typical Current Idle" turned back on, but without the "rcu_nocbs=0-15" voodoo -- so far, so good (as the man said as he fell past the 7th floor).

So still no concrete 100% sure fire solution?

Incredible.

Who knows ?

When I say that "Typical Current Idle" appears to eliminate the "freeze when idle" problem, I mean that (a) that is my experience (to date), and (b) I have seen others reporting similar experience.

I'm not aware of any public, definitive information from AMD, any motherboard vendor or the relevant kernel folk -- certainly nothing which says what the "Typical Current Idle" BIOS option actually does, and whether or how it solves the problem, in part or completely.

Sadly, this long thread has almost nothing to do with identifying any actual Kernel bug or bugs. It's more a sort of support group. [Hello, I'm Chris and I'm an AMD Ryzen user... since I found the "Typical Current Idle" BIOS option, I haven't had a "freeze while idle" for two weeks -- praise be.]

If you are looking for useful information, look away now: angry, bewildered person about to howl at the moon...

--------------

...do I find the situation "Incredible" ? hmmm...

[Final warning: reduced signal/noise ration ahead.]

I bought my Ryzen machine the moment I could order it.

When I came to build it, I found there was some confusion about the "standard" fitting for the CPU cooler -- the plate glued to the motherboard and the cooler fitting were incompatible. I found myself in a three-way stand-off between AMD, the motherboard vendor and the cooler vendor. If there was a specification for the standard AM4 socket fitting for the cooler, it was Top Secret. In the end I voided all warranties, ripped the plate from the motherboard and replaced it by the one supplied with the cooler.

When I set up my new machine, I found the standard configuration for the memory was not as fast as I expected, noting <https://www.amd.com/en/products/cpu/amd-ryzen-7-1800x>. But, I stumbled across this <https://community.amd.com/community/gaming/blog/2017/03/14/tips-for-building-a-better-amd-ryzen-system> where I discovered that the 2667 MT/s I was expecting was for 2 DIMMs; with my 4 DIMMs, 2133 MT/s is what I should have expected. Silly Me ! I have seen BIOSes since then which are supposed to improve memory support... but I think that's more options for overclocking... who knows ? Call me old fashioned, but I am disappointed (but no longer surprised) that I cannot find an AMD data sheet that specifies memory support (or much of anything else, for that matter).

Can I find any documentation for AGESA ? No. It's various versions (and changes in version numbering) ? No.

Does my motherboard vendor provide release notes for each BIOS version ? No. How did I discover which BIOS version had the "Typical Current Idle" option ? By experiment.

If the <email address hidden> fairy has died because I stopped clapping, then I'm sorry.

... so, Incredible ? Nah. Infuriating ? You bet.

Haven't had a single lockup after update to Kernel 4.15 and setting the BIOS to typical idle current setting. Have run the PC for few days and no issues nor any lock up under heavy CPU use.This is a ASUS B350M board with latest BIOS on Ryzen 1700.

I erred earlier in suggesting that this bug should be closed with
"fixed". Rather, this bug should should be closed with "notabug" because
there appears to be no evidence to suggest that it is actually a kernel
bug, compared to considerable evidence suggesting it is not.

Meanwhile, I personally appreciate the slow trickle of comments along
the lines of confirming that the typical current bios fix is effective,
compared to the absence of comments that the bug persists even with this
option enabled. My current uptime with typical current enabled is 136
days, compared to only a few days without it.

The situation is incredible, in that there doesn't (seem) to be a concrete sure fire solution to this.

If there's a BIOS option, which can fix it, it should damn well be put MANDATORY in all BIOS - not an option, locked in. System stability should not be this unreliable. Especially now they're on 2xxx series processors.

This huge job log is neatly a year old now, it's crazy. It should be fixed on the processor itself or the BIOS.

Sorry guys, typical Current Idle doesn't fix it for me.

I can't believe this is not yet nailed down. I kinda lost believe in the AMD comeback.

I do have this issue.

My setup is a 2700X on a Gigabyte B450 AORUS PRO.

I did hunt this freeze issue down to the behaviour decribed here: System is suddenly stuck during normal office use. Everything stays where is is. No restart, whoops, panic, log entries of any type.

Only reset-button helps.

I did try to diable Global C-State Control in UEFI - didn't help.
After finding this thread today I set Power Supply Idle Control to "Typical Current Idle" and set global C-State back to Auto (since I thought that doesn't help). Sadly, my system froze after 3h.

That is with the first and stock UEFI version F1 (AGESA 1.0.0.4).
I've just seen that there is an updated UEFI version F2. I've just updated and set global c-states to disabled AND Power Supply Idle Control to "Typical Current Idle".

"sudo ./zenstates.py -l" is now:
C6 State - Package - Disabled
C6 State - Core - Disabled

I'll come back with the results.

This makes the whole AMD 1xxx and 2xxx seem unstable.

Questions:
- Why does AMD not care? Is this maybe not NOT a general issue and only affecting a small percentage of users?

- If Windows doesn't suffer from that issue - can we find out why and do a "if amd then powersave like windows"?
Or is it just the lack of information/support by vendor so that we don't know what exactly the issue is, what is causing it and how to work around it?

Michael, with all that enabled also try the " idle=nomwait " kernel option If the upgrade to F2 doesn't work.

I have a similar BIOS and the same CPU. I had to enable all what you did but this wouldn't work without idle=nomwait. See my post above.

Hi, another Michael here.
The typical idle current fixed the trouble, if you have still freeze under low load after it's:
a bug in the uefi that doesn't really apply the "typical idle current"(disabling partially core parking).
another problem than the one on this thread.

I want to add another thing, about fixing it directly from the kernel.
As typical idle current seems (i am testing it again for see) not needed under the other OS, that's because the core parking under this other OS is partially disabled by DEFAULT on this OS as i said earlier in this thread.

So it seems possible to fix it via the kernel, without make use of an uefi parameters not enabled by default.
It could increase the user experience under Linux for people who doesn't know this trouble.

So, why don't make simply the same on Linux? IIRC from my reading over hardware.fr article the same apply to Intel cpu. That's a tweak for increasing performance afaik. Ryzen gains ~10%(on games, where latencies are more important) while not having "deep sleep" of cores enabled.

My Wattmeter is not a good one but i see mostly no difference between this core parking partially disabled or not. It must save really few power.

You all can easily test it by core parking control software on this other OS.

(In reply to Michaël Colignon from comment #420)
> ...
> I want to add another thing, about fixing it directly from the kernel.
> As typical idle current seems (i am testing it again for see) not needed
> under the other OS, that's because the core parking under this other OS is
> partially disabled by DEFAULT on this OS as i said earlier in this thread.
>
> So it seems possible to fix it via the kernel, without make use of an uefi
> parameters not enabled by default...

I can perhaps confirm that. But I have to say first, I'm not a developer and I do not understand anything about Linux.

I use an AMD Ryzen 5 1600 Six-Core @ 12x 3.2GHz with Gentoo as the operating system. So it's very easy for me to build a kernel just for my machine. (Without the whole Intel, ARM64, Android and Big-Data stuff.)

I try the kernels with standard bios setup, without kernel parameters like "idle = nomwait", without OC. My bios only has Global C-states Auto (which is default) and enable/disable options. There is no "Idle Control to "Typical Current Idle"" in the BIOS. Only CONFIG_RCU_NOCB_CPU=y is set in the .config.

BOOT_IMAGE=/boot/kernel-x86_64-4.19.0-rc3 root=UUID=465f9c93-09d0-441b-9440-8ec7799b557c ro real_root=/dev/sdd4 resume=/dev/sdd3 init=/usr/lib/systemd/systemd
rootfstype=ext4

./zenstates.py -l
...
C6 State - Package - Enabled
C6 State - Core - Enabled
...
On this basis,
It is to be confirmed by me that the kernel 4.17.14 works really well. But the series 4.18.x is cruel. Also the currently as "stable" declarated 4.18.7. Bad. Freeze or reboot every 2 - 3 hours.

To my surprise, the 4.19.0-rcX work well. So far no soft lockups.

I do not know why. But these are the subjective results of my playing with the kernels.

I can also understand that the few kernel developers are working on priority issues. Please think of Specter and Meltdown.

But a few clarifying information would be very welcome..

Hi,

update from me, as promised. First my system for reference:
2700X
B450 Chipset - Gigabyte B450 AORUS PRO with F2 BIOS
PSU from 2018 which is "Haswell compatible".

Typical Current Idle AND disabling global c-states didn't help.
Running Ubuntu 18.04 LTS with its stock kernel 4.15.0-34 right now.

I am having some success (4 days now) with all of that AND idle=nomwait kernel parameter - although I only read that to be related to resets and not freezes.

Anyways. will keep my system on.

I would like to trade in the LEDs on GPU and mainboard for vendors effort to get some stability in their systems.

Any update from someone who really knows what is going on would be really appreciated. We're playing around with settings which are not explained anywhere.

Displaying first 40 and last 40 comments. View all 509 comments or add a comment.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers