Ryzen 1800X freeze - rcu_sched detected stalls on CPUs/tasks

Bug #1690085 reported by Vincent
218
This bug affects 38 people
Affects Status Importance Assigned to Milestone
Linux
Expired
Medium
linux (Ubuntu)
High
Unassigned

Bug Description

Hi,

We aregetting various kernel crash on a pretty new config.
We're using Ryzen 1800X CPU with X370 Gaming Pro Carbon MB (7A32V1) using latest BIOS available (1.52)

We are running Ubuntu 17.04 (amd64), we've tried different kernel version, native one and releases from http://kernel.ubuntu.com/~kernel-ppa/mainline/ too.
Tested kernel version:

native 17.04 kernel
4.10.15

Issues are the same, we're getting random freeze on the machine.

Here is kern.log entry when happening :

May 10 22:41:56 dev2 kernel: [24366.186246] INFO: rcu_sched detected stalls on CPUs/tasks:
May 10 22:41:56 dev2 kernel: [24366.187618] 0-...: (1 GPs behind) idle=49b/1/0 softirq=28561/28563 fqs=913449
May 10 22:41:56 dev2 kernel: [24366.188977] (detected by 12, t=1860207 jiffies, g=10001, c=10000, q=4656)
May 10 22:41:56 dev2 kernel: [24366.190344] Task dump for CPU 0:
May 10 22:41:56 dev2 kernel: [24366.190345] swapper/0 R running task 0 0 0 0x00000008
May 10 22:41:56 dev2 kernel: [24366.190348] Call Trace:
May 10 22:41:56 dev2 kernel: [24366.190354] ? native_safe_halt+0x6/0x10
May 10 22:41:56 dev2 kernel: [24366.190355] ? default_idle+0x20/0xd0
May 10 22:41:56 dev2 kernel: [24366.190358] ? arch_cpu_idle+0xf/0x20
May 10 22:41:56 dev2 kernel: [24366.190360] ? default_idle_call+0x23/0x30
May 10 22:41:56 dev2 kernel: [24366.190362] ? do_idle+0x16f/0x200
May 10 22:41:56 dev2 kernel: [24366.190364] ? cpu_startup_entry+0x71/0x80
May 10 22:41:56 dev2 kernel: [24366.190366] ? rest_init+0x77/0x80
May 10 22:41:56 dev2 kernel: [24366.190368] ? start_kernel+0x464/0x485
May 10 22:41:56 dev2 kernel: [24366.190369] ? early_idt_handler_array+0x120/0x120
May 10 22:41:56 dev2 kernel: [24366.190371] ? x86_64_start_reservations+0x24/0x26
May 10 22:41:56 dev2 kernel: [24366.190372] ? x86_64_start_kernel+0x14d/0x170
May 10 22:41:56 dev2 kernel: [24366.190373] ? start_cpu+0x14/0x14
May 10 22:44:56 dev2 kernel: [24546.188093] INFO: rcu_sched detected stalls on CPUs/tasks:
May 10 22:44:56 dev2 kernel: [24546.189461] 0-...: (1 GPs behind) idle=49b/1/0 softirq=28561/28563 fqs=935027
May 10 22:44:56 dev2 kernel: [24546.190823] (detected by 14, t=1905212 jiffies, g=10001, c=10000, q=4740)
May 10 22:44:56 dev2 kernel: [24546.192191] Task dump for CPU 0:
May 10 22:44:56 dev2 kernel: [24546.192192] swapper/0 R running task 0 0 0 0x00000008
May 10 22:44:56 dev2 kernel: [24546.192195] Call Trace:
May 10 22:44:56 dev2 kernel: [24546.192199] ? native_safe_halt+0x6/0x10
May 10 22:44:56 dev2 kernel: [24546.192201] ? default_idle+0x20/0xd0
May 10 22:44:56 dev2 kernel: [24546.192203] ? arch_cpu_idle+0xf/0x20
May 10 22:44:56 dev2 kernel: [24546.192204] ? default_idle_call+0x23/0x30
May 10 22:44:56 dev2 kernel: [24546.192206] ? do_idle+0x16f/0x200
May 10 22:44:56 dev2 kernel: [24546.192208] ? cpu_startup_entry+0x71/0x80
May 10 22:44:56 dev2 kernel: [24546.192210] ? rest_init+0x77/0x80
May 10 22:44:56 dev2 kernel: [24546.192211] ? start_kernel+0x464/0x485
May 10 22:44:56 dev2 kernel: [24546.192213] ? early_idt_handler_array+0x120/0x120
May 10 22:44:56 dev2 kernel: [24546.192214] ? x86_64_start_reservations+0x24/0x26
May 10 22:44:56 dev2 kernel: [24546.192215] ? x86_64_start_kernel+0x14d/0x170
May 10 22:44:56 dev2 kernel: [24546.192217] ? start_cpu+0x14/0x14

Depending on the kernel version, we've got NMI watchdog errors related to CPU stuck (mentioning the CPU core id, which is random).
Crash is happening randomly, but in general after some hours (3-4h).

Now, we've installed kernel 4.11.0-041100-generic #201705041534 this morning and waiting for crash...
For now, the machine is not "used", at least, it's not CPU stressed...

Thanks
---
ApportVersion: 2.20.4-0ubuntu4
Architecture: amd64
DistroRelease: Ubuntu 17.04
InstallationDate: Installed on 2017-05-09 (1 days ago)
InstallationMedia: Ubuntu-Server 17.04 "Zesty Zapus" - Release amd64 (20170412)
Package: linux (not installed)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=fr_FR.UTF-8
 SHELL=/bin/bash
Tags: zesty
Uname: Linux 4.11.0-041100-generic x86_64
UnreportableReason: The running kernel is not an Ubuntu kernel
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:

_MarkForUpload: True

Revision history for this message
Vincent (hvincent13) wrote :
Revision history for this message
Brad Figg (brad-figg) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1690085

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Vincent (hvincent13) wrote : JournalErrors.txt

apport information

tags: added: apport-collected zesty
description: updated
Revision history for this message
Vincent (hvincent13) wrote : ProcCpuinfoMinimal.txt

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Lipido (lipido) wrote :

Hi Vicent,

Did you experience more crashes with kernel 4.11?

Thank you!

Revision history for this message
Vincent (hvincent13) wrote :

Hi Lipido,

Yes, i'm still experiencing crashes, even when using 4.11

Please find full kern.log in attachment.

Regards,

Revision history for this message
Lipido (lipido) wrote :

Ok, me too :-(

Crashes appear from 24-48h of operation.

My hardware is:
- SSUS PRIME B350-Plus
- Amd Ryzen 5 1600

OS: Ubuntu 16.04

Tried:
- Update BIOS to the latest (v 609)
- Kernel 4.10
- Disable SMT in bios (from 12 threads to 6 threads)
- Boot clocksource=tsc iommu=soft
- Disable IOMMU in bios

None of these work. I was waiting for 4.11 to be available for Ubuntu as an official package, but it seems that this will not work either.

Another link talking about what seems to be the same issue:

https://forum.level1techs.com/t/ryzen-vs-ubuntu/115715/22

Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.12 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.12-rc1/

Changed in linux (Ubuntu):
importance: Undecided → High
status: Confirmed → Incomplete
tags: added: kernel-da-key
Revision history for this message
Vincent (hvincent13) wrote :

Hi Joseph,

I've just installed 4.12-rc1.
Now waiting for a crash... or not (hope so !)

Regards,

Revision history for this message
Vincent (hvincent13) wrote :

Just go a kernel panic...

How to add the tag kernel-bug-exists-upstream ?

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
tags: added: kernel-bug-exists-upstream
Revision history for this message
Joseph Salisbury (jsalisbury) wrote :

This issue appears to be an upstream bug, since you tested the latest upstream kernel. Would it be possible for you to open an upstream bug report[0]? That will allow the upstream Developers to examine the issue, and may provide a quicker resolution to the bug.

Please follow the instructions on the wiki page[0]. The first step is to email the appropriate mailing list. If no response is received, then a bug may be opened on bugzilla.kernel.org.

Once this bug is reported upstream, please add the tag: 'kernel-bug-reported-upstream'.

[0] https://wiki.ubuntu.com/Bugs/Upstream/kernel

Changed in linux (Ubuntu):
status: Confirmed → Triaged
Revision history for this message
Vincent (hvincent13) wrote :

Hi,

Could you please tell me which address I have to send the mail to?
I don't really understand how to achieve this bug report on the mailing list.

Thanks,

Revision history for this message
Vincent (hvincent13) wrote :

Hi,

It looks like that the system is stable when removing "nouveau" driver.
Waiting 24/48h and will post again.

Regards,

Revision history for this message
camparijet (iichikolamp) wrote :

Hi Vincent,

I also met the problem, and your advice removing "nouveau" succeeded for work-around to stabilize system. Before it freeze every 3-5hr after booting up, but now working without problem for 2 days.

My enviroment is:
- cpu: Ryzen 1700
- motherboard: ASUS B350M-A
- graphics card: NVIDIA GK208
- kernel: 4.11.0-041100rc8-generic
- dist: 17.04

Revision history for this message
Vincent (hvincent13) wrote :

Hi,

5 days update now and no crash.

camparijet => did you installed NVIDIA driver instead ?

Regards,

Revision history for this message
camparijet (iichikolamp) wrote :

Hi Vincent,

> camparijet => did you installed NVIDIA driver instead ?

No. I don't have to use the card for my purpose, so simply i disable it.

Revision history for this message
Alex Jones (blenheimears) wrote :

I'm seeing this crash even with the Nvidia official driver.

Revision history for this message
Alex Jones (blenheimears) wrote :

This is a hardware bug in the CPU. This ticket should be closed as invalid.

Changed in linux (Ubuntu):
status: Triaged → Invalid
Revision history for this message
Alex Jones (blenheimears) wrote :

Reopening because even though this is a known issue with the CPU we could still implement a workaround. One workaround is to disable address space layout randomization:

echo 0 >/proc/sys/kernel/randomize_va_space

However, that would be disabling a security feature.

Changed in linux (Ubuntu):
status: Invalid → New
Revision history for this message
Joseph Salisbury (jsalisbury) wrote : Status changed to Confirmed

This change was made by a bot.

Changed in linux (Ubuntu):
status: New → Confirmed
tags: added: artful
Revision history for this message
Marc Rene Schädler (suaefar) wrote :

Was this issue officially confirmed to be a hardware bug in Ryzen processors by AMD?
If so, could you provide a link to the statement?

Disabling address space layout randomization (ASLR) seems to alleviate the problem, but does it solve it?

I am investigating unstable behavior under load which could be related.
There, disabling ASLR is not sufficient!
People suggest to increase SOC voltages, use specific versions of the kernel and the like.
See https://community.amd.com/thread/215773?start=135&tstart=0 for more info.

Revision history for this message
Alex Jones (blenheimears) wrote :

AMD has not publicly commented on this issue that I'm aware of. This issue has been seen on many different operating systems. DragonFlyBSD includes a workaround for this issue. The workaround on Linux is to compile the kernel with CONFIG_RCU_NOCB_CPU, CONFIG_RCU_NOCB_CPU_ALL, and disable ASLR using "echo 0 >/proc/sys/kernel/randomize_va_space". This can be put into rc.local. It's also possible to hardcode ASLR as disabled into the kernel, but this requires modifying the kernel source, not just the config file. There is a new AGESA update released about a week ago, 1.0.0.6a, although I have not tested whether the new AGESA alone (without any kernel changes) solves the issue.

Revision history for this message
Alex Jones (blenheimears) wrote :

I just tested 1.0.0.6a AGESA, and it does not solve this issue. In some cases just one program will crash, and in other cases the entire system will crash. I will test the workaround above later.

Revision history for this message
Alex Jones (blenheimears) wrote :

If any of your RAM timings are odd (eg. 17), setting them to the next even number (eg. 18) helps a lot. Recompiling the kernel with CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_ALL, and disabling ASLR is still necessary though. It may also be a good idea to give the SOC slightly more voltage, but not more than 1.2 V.

Revision history for this message
Kai-Heng Feng (kaihengfeng) wrote :

Alex,

Can you provide link on how DragonflyBSD fixed this issue?

Revision history for this message
Alex Jones (blenheimears) wrote :

This is the DragonFlyBSD commit. https://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/b48dd28447fc8ef62fbc963accd301557fd9ac20

It appears that there are two different ways that the system can crash, which is why it is necessary to both disable ASLR and to compile the kernel with CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_ALL. The ASLR-related crash usually results in a single or a few programs crashing (although if an important program crashes it can bring down the entire system) and happens under heavy load. The other crash only happens if the kernel was compiled without CONFIG_RCU_NOCB_CPU and CONFIG_RCU_NOCB_CPU_ALL (Ubuntu's kernel is compiled without these options) and happens when the system is idle or nearly idle, and results in a complete system crash.

Revision history for this message
Alex Jones (blenheimears) wrote :

If the motherboard allows it, disabling the OpCache will completely prevent (or at least greatly reduce the probability of) the ASLR-related crash, even if ASLR is enabled in the kernel. As far as I'm aware it has no effect on the other type of crash.

Revision history for this message
Alex Jones (blenheimears) wrote :

I've also determined that changing the CPU, memory, or SOC voltages or timings has little or no effect on either type of crash.

Revision history for this message
Kai-Heng Feng (kaihengfeng) wrote :

This is beyond my expertise - let's see what upstream can do.

Revision history for this message
Torge (cyslider) wrote :

I also experience this problem since I updated yesterday.

I am using Kubuntu 16.04 with KDE backports enabled.
I also have a Ryzen 1800X and an Asrock X370 Gaming professional

I experience this either when I boot and don't log in promptly or when I enter the lockscreen.

After reading this page I tried to run
  seq 1 | xargs -P0 -n1 md5sum /dev/zero &

as root to always keep one CPU core busy and indeed this seems to prevent the lockscreen problem...

I would also like to note that the system was not 100% frozen, once every few minutes it seemed to response for a short time, enabling me to switch to the TTY. Then I always saw some processes hanging at 100% that I could not kill, or the kill was delayed for a long time. I always see the errors in the initial posts during that time. The TTY seemes to work fine though, once entered though.

Revision history for this message
David Wilson (plottt) wrote :

I've ran into this problem several times. After disabling C-states in the motherboard, my system has been running stable for ~5 weeks uptime so far.

Revision history for this message
Torge (cyslider) wrote :

Ok, this trick did not help and my PC did not make it through the night without freezing again.

However I finally found the real problem. I installed oibaf a while back as my Kubuntu was flickering all over the place. Now it seems the be the source of my problem. I deinstalled oibaf again and now everything seems fine. No lock screen freezing for me anymore.

Revision history for this message
Stuart Page (sdpagent) wrote :

Just wanting to report that I am also experiencing this issue with a freshly installed Ubuntu Server 16.04.3 with 4.10 kernel on the 22nd August 2017. All the updates applied and running as a KVM host with hardware:

Ryzen 1700 (non-x)
Motherboard: Asus prime B350-Plus
Bios: - Version 0805
 - Disabled SMT
 - Disabled c-states

Revision history for this message
xb5i7o (xb5i7o) wrote :

Hi guys,

Im getting the same issues, brand new build.

Ryzen 1800x
Asrock X370 Taichi
BIOS: 3.10 (latest)
I tried disabling cool n quiet and c-state

However there is another option under advanced - for GLOBAL C-States that i just disabled today and i am waiting to see what will happen.

On BIOS v3.00 PC would freeze after 6 hours of idling or not touching anything. Come back to see my keyboard and mouse and everything was frozen.
With BIOS v.3.10 now i get random reboots atleast once a day.

Im running Ubuntu 16.04.3 Kernel 4.4.0-93

Anyone know to direct me where the SoC voltage would be in BIOS? is it the VDD SoC?
Anyone know how to update my kernel to 4.10 if it tells me i have the latest?

Revision history for this message
Stuart Page (sdpagent) wrote :

I finally managed to figure out how to compile a kernel with RCU_NOCB and disabled ASLR as Alex Jones mentioned, and it appears to have worked for me and another guy who helped me put the tutorial together:

http://blog.programster.org/ubuntu-16-04-compile-custom-kernel-for-ryzen

Revision history for this message
Kai-Heng Feng (kaihengfeng) wrote :

Stuart,

Does this issue also happen on latest mainline kernel?

Revision history for this message
Franck Charras (franckc) wrote :

Hi,
I'm getting the same issues on several identical builds, with an ASUS prime X370-PRO motherboard.
It's very hard to analyze since it happens randomly every other week, and it leaves no logs. It seems that the freeze happens at idle after very high memory load (observation after logging CPU and RAM loads).
Compiling the kernel as suggested in this thread didn't work (new freeze this morning).

Revision history for this message
tgui (eric-c-morgan) wrote :

I too still have random computer shutdowns without logs. Uptime varies from a couple days to a week or so. It happens it seems after being relatively idle for a long period. I do have high memory usage because of the VMs I run.

I've disabled C-states, cool and quiet, tested memory, and so forth. I also compiled a new kernel as mentioned by Stuart. I do not have segfaults with compilations.

I am willing to run tests and provide info. Please let me know if anyone wants something.

Ubuntu 16.04
Asrock x370 ITX
32GB Ram
Ryzen 1700

Revision history for this message
Franck Charras (franckc) wrote :

Ryzen CPUs manufactured before week 24 of this year were known to have issues (especially the segfault issue). It was officially fixed for all ryzen manufactured after week 30. All my ryzen are pre-24 CPUs and they all have this silent crash issue. Does it also happen with the post week 30 ryzen ? (@tgui what is yours ?)

Revision history for this message
AS (as2008) wrote :

I can confirm the issue with a week 33 Ryzen 1700 on Asus Prime B350 Plus.
I had segfaults with my previous 1700 (week 22 iirc). No more segfaults with the week 33 CPU but still random crashes on (long time) idle.

tags: removed: kernel-da-key
information type: Public → Public Security
information type: Public Security → Public
Changed in linux:
importance: Unknown → Medium
status: Unknown → Confirmed
Brad Figg (brad-figg)
tags: added: cscc
Changed in linux:
status: Confirmed → Expired
759 comments hidden view all 833 comments
Revision history for this message
In , ewerton.urias (ewerton.urias-linux-kernel-bugs) wrote :

Hello everyone.

I apologize for my English, I'll try to communicate.

I did a hardware upgrade in November 2019 (Intel for AMD), my current hardware is this:

------------------------------------------------
ASUS TUF B450-PRO GAMING
Ryzen 5 1600 Six-Core (BIOS always updated)
GeForce GTX 960 4 GB 128 Bits
Corsair 650w
Corsair LPX 16 GB 2666
------------------------------------------------

During the first few weeks, I noticed reboots and freezes, and after a few months of research, I found an alternative solution, which is just to add "processor.max_cstate=1" to Grub.

After I did this, my computer went 6 months without rebooting and/or freezing.

Yesterday I removed this parameter from Grub, to see what the result would be, and it happened, I had a reboot and a freeze, it means that "processor.max_cstate=1" is a solution for me.

The reason I'm here is to understand the root of the problem, to correct it correctly, I'll soon test "Power Supply Idle Control" and "Global C-State Control" in BIOS, but I have seen that for some users it didn't work.

I'm trying to read all of your comments (there are many), but skipping to the last comments, it seems that there is still no solution to this problem, and this makes me very sad.

I don't know anything about hardware, could someone explain to me, in a layman's way, the difference between using "processor.max_cstate=1", "processor.max_cstate=5", "Power Supply Idle Control" and "Global C-State Control"?

I thank you for your patience.

Revision history for this message
In , ashesh.ambasta (ashesh.ambasta-linux-kernel-bugs) wrote :

I can confirm that in my case, all the suggested alternatives in this
thread didn't work (the ones that were applicable to my use-case anyway).

In the end, I threw my hands up and did an RMA. And that was over a
month ago. It seems to have solved the isssue. The processor I had
previously and its replacement were different batches; the former being
from 2019 and the latter from Feb. 2020. It seems to me that AMD ironed
out some issues with processors and I got lucky with the replacement. Or
it could just be some tiny variations in fabrication. I can never be sure.

But my machine has been up since the RMA without any crashes.

On 9/17/20 7:25 AM, <email address hidden> wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=196683
>
> Ewerton Urias (<email address hidden>) changed:
>
> What |Removed |Added
> ----------------------------------------------------------------------------
> CC| |<email address hidden>
>
> --- Comment #719 from Ewerton Urias (<email address hidden>) ---
> Hello everyone.
>
> I apologize for my English, I'll try to communicate.
>
> I did a hardware upgrade in November 2019 (Intel for AMD), my current
> hardware
> is this:
>
> ------------------------------------------------
> ASUS TUF B450-PRO GAMING
> Ryzen 5 1600 Six-Core (BIOS always updated)
> GeForce GTX 960 4 GB 128 Bits
> Corsair 650w
> Corsair LPX 16 GB 2666
> ------------------------------------------------
>
> During the first few weeks, I noticed reboots and freezes, and after a few
> months of research, I found an alternative solution, which is just to add
> "processor.max_cstate=1" to Grub.
>
> After I did this, my computer went 6 months without rebooting and/or
> freezing.
>
> Yesterday I removed this parameter from Grub, to see what the result would
> be,
> and it happened, I had a reboot and a freeze, it means that
> "processor.max_cstate=1" is a solution for me.
>
> The reason I'm here is to understand the root of the problem, to correct it
> correctly, I'll soon test "Power Supply Idle Control" and "Global C-State
> Control" in BIOS, but I have seen that for some users it didn't work.
>
> I'm trying to read all of your comments (there are many), but skipping to the
> last comments, it seems that there is still no solution to this problem, and
> this makes me very sad.
>
> I don't know anything about hardware, could someone explain to me, in a
> layman's way, the difference between using "processor.max_cstate=1",
> "processor.max_cstate=5", "Power Supply Idle Control" and "Global C-State
> Control"?
>
> I thank you for your patience.
>

Revision history for this message
In , pmenzel+bugzilla.kernel.org (pmenzel+bugzilla.kernel.org-linux-kernel-bugs) wrote :

(In reply to Ashesh Ambasta from comment #720)
> I can confirm that in my case, all the suggested alternatives in this
> thread didn't work (the ones that were applicable to my use-case anyway).
>
> In the end, I threw my hands up and did an RMA. And that was over a
> month ago. It seems to have solved the isssue. The processor I had
> previously and its replacement were different batches; the former being
> from 2019 and the latter from Feb. 2020. It seems to me that AMD ironed
> out some issues with processors and I got lucky with the replacement. Or
> it could just be some tiny variations in fabrication. I can never be sure.
>
> But my machine has been up since the RMA without any crashes.

As you disassembled the processor, were you able to copy the serial numbers from the old and new one, and could post them here please? Could you pleas

PS: Please always remove the full quote from messages, when replying by email, as it just clutters the Web interface.

Revision history for this message
In , qdeqde (qh-u) wrote :

my R5 3600 freezing was solved by RMA, running 3 weeks without problem

old serial with freezing : Y939866S90122
current serial running OK: 9JC2491R00354

Revision history for this message
Nigel Reed (nelgin) wrote :

Another Ryzen 7 1800X user here with similar random crashes.

[165716.089703] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[165716.095949] rcu: 1-...!: (0 ticks this GP) idle=354/0/0x0 softirq=2154363/2154363 fqs=0
[165716.104512] rcu: 3-...!: (0 ticks this GP) idle=29c/0/0x0 softirq=883832/883832 fqs=0
[165716.112873] rcu: 4-...!: (8 GPs behind) idle=ad8/0/0x0 softirq=2165586/2165586 fqs=0
[165716.121179] rcu: 9-...!: (9 GPs behind) idle=acc/0/0x0 softirq=1340600/1340600 fqs=0
[165716.129467] rcu: 11-...!: (2 GPs behind) idle=a18/0/0x0 softirq=4538536/4538537 fqs=0
[165716.137828] rcu: 12-...!: (0 ticks this GP) idle=870/0/0x0 softirq=2158040/2158040 fqs=0
[165775.697763] rcu: rcu_sched kthread starved for 29898 jiffies! g36134941 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=3
[165775.709013] rcu: RCU grace-period kthread stack dump:
[165837.494623] watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [(resolved):52315]
[165865.494840] watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [(resolved):52315]
[165893.495058] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [(resolved):52315]
[165921.495276] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [(resolved):52315]
[165949.495494] watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [(resolved):52315]

Unfortunaltey I didn't have sysrq configured so I can't provide more information other than I also have an X370 motherboard.

The system is not not has it ever been overlocked. I've owned all the componentes since new so I'm not going to be able to RMA them like some others have.

1 comments hidden view all 833 comments
Revision history for this message
In , ashesh.ambasta (ashesh.ambasta-linux-kernel-bugs) wrote :

Apologies for the late reply.
> As you disassembled the processor, were you able to copy the serial numbers
I did copy and send the older processor's serial numbers to the dealer,
for the RMA. That was a processor from a batch in late 2018 IIRC.
> from the old and new one, and could post them here please? Could you pleas
The new one comes from a batch in Feb. of 2020. I'm afraid I don't have
the serial number copied somewhere (it is probably on the box but that
doesn't show the batch number). That was a rookie mistake.
> PS: Please always remove the full quote from messages, when replying by
> email,
> as it just clutters the Web interface.
Noted, sorry about that!

Revision history for this message
In , mclark (mclark-linux-kernel-bugs) wrote :

if I disable multithreading my system that had these crashes is stable for months without them. e.g. run this in rc.local:

#!/bin/bash
#
# disables hyperthreading, which stops the system from crashing
#
   for CPU in /sys/devices/system/cpu/cpu[0-9]*; do
        CPUID=$(basename $CPU)
        echo "CPU: $CPUID";
        if test -e $CPU/online; then
                echo "1" > $CPU/online;
        fi;

        COREID="$(cat $CPU/topology/core_id)";
        eval "COREENABLE=\"\${core${COREID}enable}\"";

        if ${COREENABLE:-true}; then
                echo "${CPU} core=${CORE} -> enable"
                eval "core${COREID}enable='false'";
        else
                echo "$CPU core=${CORE} -> disable";
                echo "0" > "$CPU/online";
        fi;
    done;

Revision history for this message
In , awwit (awwit-linux-kernel-bugs) wrote :

Hello everyone. I've been following this topic for a long time. I have the same problem on my Rizen 1700. For a while, I was helped by switching "Power Supply Idle Control" to "Typical Current Idle". For a while, everything worked fine for me with the value "auto". But then I changed something in the settings of my BIOS and the lockup began to happen even with the value "Typical Current Idle".

Motherboard: GA-AB350N-Gaming WIFI (rev. 1.0)
Linux Kernel Version: 5.9.12

Then I realized that the bug is due to a combination of some BIOS settings.

So far, I have managed to achieve stable operation with the following settings:

"Power Supply Idle Control" -> "Typical Current Idle"
"Power" (Tab) -> "ErP" -> "Enabled"
"Power" (Tab) -> "CEC 2019 Ready" -> "Disabled"

The settings on the "Power" tab are very important! Especially "CEC 2019 Ready (Enabled)" setting, with which lockups are repeated.

I will try to change different BIOS settings to achieve stable system operation when the "Power Supply Idle Control" is set to "auto".

Revision history for this message
In , ewerton.urias (ewerton.urias-linux-kernel-bugs) wrote :

I'm here again, after my first comment (Comment 719).

I have Ryzen 5 1600 (UA 1843PGS), and I come to report some news about my case.

I have 2x8GB 2666 Mhz, but in the BIOS Setup, by default, the frequency is set to "AUTO" (2133 Mhz), and in 2019 I had applied it to 2666 Mhz, and I didn't remember this.

My friend warned me about this, that what I did was an "overclocking of RAM", I didn't know that this is an overclocking, so I decided to revert, applying "AUTO" again (2133 Mhz).

Since then... the reboots have stopped, however... the system has continued to freeze, and the logs point only to my NVIDIA GTX 960.

In the BIOS Setup, I disabled the "Global C-State Control", and since then, it has been almost 60 days since there are no more system freezes or reboots, things that happened every day.

Resume:

1. Maintaining RAM frequency in AUTO resolves reboots;
2. Disabling "Global C-State Control" solves my problem with system freezes (log points to NVIDIA).

These two things above solved my problem, and that parameter "processor.max_cstate = 1" in GRUB is no longer needed.

Revision history for this message
In , nelsoneci (nelsoneci-linux-kernel-bugs) wrote :

Created attachment 294183
attachment-20815-0.html

Hi Ewerton.

This might be a side effect and kind of unrelated with the bug. Since YMMV
and this can wreck your HW please read this as an anecdotal experience.
When you overclock RAM you need to change other stuff in order to stabilize
the system. This video from AMD people helped me get 3000Mhz with no
issues. Before this video I was wasting -part of- the money I paid for the
sticks running at 2666. This was with an Ab350m-ds3h (BIOS F31) and a Ryzen
2700X.

  https://www.youtube.com/watch?v=vZgpHTaQ10k

BTW, Memtest86 helped me a lot here. It would detect an invalid
configuration in a second. I was happy after it ran a few times and I
didn't have any issues after that. Since then I switched to a recent
motherboard and the RAM was detected automatically.

On Wed, Dec 16, 2020 at 9:22 PM <email address hidden> wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=196683
>
> --- Comment #726 from Ewerton Urias (<email address hidden>) ---
> I'm here again, after my first comment (Comment 719).
>
> I have Ryzen 5 1600 (UA 1843PGS), and I come to report some news about my
> case.
>
> I have 2x8GB 2666 Mhz, but in the BIOS Setup, by default, the frequency is
> set
> to "AUTO" (2133 Mhz), and in 2019 I had applied it to 2666 Mhz, and I
> didn't
> remember this.
>
> My friend warned me about this, that what I did was an "overclocking of
> RAM", I
> didn't know that this is an overclocking, so I decided to revert, applying
> "AUTO" again (2133 Mhz).
>
> Since then... the reboots have stopped, however... the system has
> continued to
> freeze, and the logs point only to my NVIDIA GTX 960.
>
> In the BIOS Setup, I disabled the "Global C-State Control", and since
> then, it
> has been almost 60 days since there are no more system freezes or reboots,
> things that happened every day.
>
> Resume:
>
> 1. Maintaining RAM frequency in AUTO resolves reboots;
> 2. Disabling "Global C-State Control" solves my problem with system freezes
> (log points to NVIDIA).
>
> These two things above solved my problem, and that parameter
> "processor.max_cstate = 1" in GRUB is no longer needed.
>
> --
> You are receiving this mail because:
> You are on the CC list for the bug.

Revision history for this message
In , kallisti5 (kallisti5-linux-kernel-bugs) wrote :

Some information. I was having this issue all the time on my Asus x370 and my Ryzen 1800x.

I upgraded my mainboard to an Asus x470, and the issue disappeared overnight. Same ram/CPU/add-on cards/OS

So, potentally due to the x370, or the combonation of early Ryzen + x370

Revision history for this message
In , ashesh.ambasta (ashesh.ambasta-linux-kernel-bugs) wrote :

So after months after my RMA, this issue seems to have surfaced again. I
experienced a lockup on system idle just a while ago.

It seems to also somehow be related to particular boots: my system was
off (not suspend, powered down) last night. It had been stable for
months before that.

I'll be selling my 2950/mobo/cooler and switching to a more stable Intel
(something old). Life is too short to live on the "edge" with these AMD
processors.

Revision history for this message
In , raulvior.bcn (raulvior.bcn-linux-kernel-bugs) wrote :

This thing can happen due to multiple factors.
I was running a 1800X. Freezes ocurred in 24-48h of uptime. Disabling Global C-State or enabling typical power idle in UEFI stopped those freezes. The latter option disables PC6 on top of using 0.85V idle voltage. To disable PC6 you need a cold boot, otherwise only the voltage change is applied and PC6 remains enabled, crashing the system as usual.

I upgraded to a 2700X (UEFI cleared) and apparently the issue disappeared. But nope, it's just less frequent. Much less. Currently I have an uptime of 23d. I also had another 2700X running for 17d before testing again a 1800X which crashed within 24h, afterwards I inserted the current 2700X which crashed within 48h. The following boot is the one with an uptime of 23d.

All in all, this might be related to either a PSU thing or to a non-existent but required kernel workaround for a bug in the processor, as detailed in the AMD Revision Guide [1]. The 1800X would crash more because it has more bugs and workarounds needed, while the 2700X has fewer, specially related to the PCIe controller. Ironically, the 2700X consumes less power at idle than the 1800X because it requires lower voltages: 12nm+ vs 14nm, and the voltages specified at every power level are also lower. I don't see much logic in saying that the PSU is the culprit of system instability.

All the people trying "idle=nomwait", "idle=halt" or "processor.max_cstate=5" should be warned those options are useless. There are only 2 Cstates available in Ryzen systems, so if you want to limit Cstate you have to set it to 1 at most -> "processor.max_cstate=1". And the use of the MWAIT instruction is disabled by the UEFI if you insert a Ryzen 1800X processor. The 2700X and the rest of 2nd gen Ryzen are not affected by any MWAIT bug. Idle option is thus useless too.

It's better to try with "pcie_aspm=off" or "pcie_aspm=force" pcie_aspm.policy=performance" and or "nvme_core.default_ps_max_latency_us=0". Maybe the PCIe root or anything related does not wakeup and the processor stalls waiting for an interrupt to be served.

[1] https://www.amd.com/system/files/TechDocs/55449_Fam_17h_M_00h-0Fh_Rev_Guide.pdf

Revision history for this message
In , ricki (ricki-linux-kernel-bugs) wrote :

I have a Ryzen 1700X and I'm heavily affected by the problem. Default kernel parameters: lockups every 1-2 days. With "processor.max_cstate=5 rcu_nocbs=0-15": reboot instead of lockups, but less frequent. Still too cumbersome to deal with.

Does anyone know whether this problem also occurs with Ryzen 5? I'd like to give it a second chance.

Revision history for this message
In , ashesh.ambasta (ashesh.ambasta-linux-kernel-bugs) wrote :

I did some research into the latest Ryzens, and I suspect the issue is
still not solved. I did manage to read some alarming posts discussing
similar lockups.

I suspect this also has to do with the motherboard. At this stage, I'm
no longer willing to try (I've done one RMA and 2.5 years with an
unreliable system).

Not to rant, but I had a similar experience with AMD a decade earlier.
Maybe its time for me to move on. :-)

Revision history for this message
In , exander77 (exander77-linux-kernel-bugs) wrote :

As far as I can tell it was solved for me a quite some time ago (on my A485), most probably a by a Bios update (with previously employed kernel updates), and I have not encountered it E459 with newer Ryzen at all.

I have never encountered it on Windows.

My thinking is that it was a combination of kernel behaviour with some in early BIOSes and was fixed when both were updated.

If you use Desktop and you have latest BIOS I would suggest replacing the motherboard as somebody pointed above.

Revision history for this message
In , ashesh.ambasta (ashesh.ambasta-linux-kernel-bugs) wrote :

Created attachment 294817
attachment-13720-0.html

After some more digging around, I see the following in `journalctl -b-1` :

    Jan 23 14:48:41 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3: Mismatch between completed Set TR Deq Ptr command & xHCI internal state.
    Jan 23 14:48:41 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3: ep deq seg = 00000000734e9686, deq ptr = 00000000f203822a

and indeed, the "crashes" I experience also seem to be accompanied by
all my USB devices (mouse, keyboard) losing power.

Revision history for this message
In , ashesh.ambasta (ashesh.ambasta-linux-kernel-bugs) wrote :

Created attachment 294819
attachment-29532-0.html

Actually, never mind.

I also see the following when looking at all similar messages:

    |╰─$ journalctl| grep 'xhci.*Mismatch' ||
    ||Jun 13 14:43:09 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Jun 27 17:58:56 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Jul 16 18:10:53 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Jul 19 16:53:52 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Sep 10 12:24:53 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Oct 09 13:23:53 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Oct 10 12:40:37 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Nov 01 16:44:36 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Dec 21 12:24:03 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Jan 13 19:09:51 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.||
    ||Jan 23 14:48:41 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
    Mismatch between completed Set TR Deq Ptr command & xHCI internal
    state.|

There were no crashes in December/November/October & neither did I lose
any USB functionality as I did now.

In any case, I'm still looking to replace this hardware.

Revision history for this message
In , pmenzel+bugzilla.kernel.org (pmenzel+bugzilla.kernel.org-linux-kernel-bugs) wrote :

(In reply to Ashesh Ambasta from comment #736)

[…]

> I also see the following when looking at all similar messages:
>
> |╰─$ journalctl| grep 'xhci.*Mismatch' ||
> ||Jun 13 14:43:09 quasar-nixos-tr kernel: xhci_hcd 0000:09:00.3:
> Mismatch between completed Set TR Deq Ptr command & xHCI internal
> state.||

[…]

Please report that to the Linux USB folks (entry for USB SUBSYSTEM in the file MAINTAINERS [1]). If you do, please also attach output of `lsusb`.

[1]: https://www.kernel.org/doc/linux/MAINTAINERS

Revision history for this message
In , jvdelisle (jvdelisle-linux-kernel-bugs) wrote :

Comment 743 is Phishing, do not click links.

Revision history for this message
In , karthik.gana12 (karthik.gana12-linux-kernel-bugs) wrote :

<a href=https://www.infobrez.com/GST-software">GST Software</a>
EasemyGST is a cloud-based gst billing software for enterprises. It has a complete tax management feature that helps in generating gst invoices and file GST returns. This gst filing software enables efficient accounting and tax management along with the below features.

https://www.infobrez.com/GST-software

Revision history for this message
In , chandrunan90 (chandrunan90-linux-kernel-bugs) wrote :
Revision history for this message
In , amirthalingam013 (amirthalingam013-linux-kernel-bugs) wrote :

Thanks for raising this question, myself too face the same.
https://www.infobrez.com/Payroll-software

Revision history for this message
In , agmondroid (agmondroid-linux-kernel-bugs) wrote :

Hello,

after while with random AMD Ryzen 5 reboots, final solution was add Kernel parameter:

acpi_osi=Linux

3 days of stability with Ubuntu 18.04 5.4.0-80-generic and counting.

Revision history for this message
In , pmenzel+bugzilla.kernel.org (pmenzel+bugzilla.kernel.org-linux-kernel-bugs) wrote :

Agmon, thank you for your report. Please open a separate issue with the full output of `dmesg` attached, and the output of `acpidump`. Please reference the issues here.

Revision history for this message
In , transglobalaarav (transglobalaarav-linux-kernel-bugs) wrote :

"Transglobal Overseas Education Consultants is the leading Overseas Consultants in Delhi since 2010. We are providing Study visas, PR Visa, Tourist Visa, Spouse Visa with free admission guidance and free counseling. We provide Our services in UK, USA, Canada, Australia, New Zealand, Singapore. Our head office in Tilak Nagar and Branch Office is Banglore & Ahmedabad. We place the right student in the right universities."
<a href="https://www.google.com/maps/place/Transglobal+Overseas+Education+Consultants/@28.6355308,77.0949389,17z/data=!3m1!4b1!4m5!3m4!1s0x390d049f4c17f821:0x53dd7da5bbf54bf!8m2!3d28.6355261!4d77.0971276">overseas education consultants in delhi</a>

<a href="https://www.google.com/maps/place/Transglobal+Overseas+Education+Consultants/@28.6355308,77.0949389,17z/data=!3m1!4b1!4m5!3m4!1s0x390d049f4c17f821:0x53dd7da5bbf54bf!8m2!3d28.6355261!4d77.0971276">abroad education consultants in delhi</a>

Revision history for this message
In , ajlopies (ajlopies-linux-kernel-bugs) wrote :
Revision history for this message
In , arun123seo (arun123seo-linux-kernel-bugs) wrote :

thanks for the information..

https://www.123coimbatore.com/

Revision history for this message
In , arun123seo (arun123seo-linux-kernel-bugs) wrote :
Revision history for this message
In , forerunsoftwaresolutions (forerunsoftwaresolutions-linux-kernel-bugs) wrote :
Download full text (3.5 KiB)

very interesting article ! more informative thanks for sharing..
<a href="http://www.forerunsoftwaresolutions.com/">Software Training Institute in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/adobe-experience-manager-training-course-in-chennai.php">Best AEM Training Institute In Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/iot-training-course-in-chennai.php">IOT Online Course in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/react-native-training-course-in-chennai.php">React Native Training Classes in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/iot-training-course-in-chennai.php">Internet of things Academy in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/ios-training-course-in-chennai.php">IOS Course in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/laravel-training-course-in-chennai.php">Php Coaching Centre in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/android-training-course-in-chennai.php">Android Training Course in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/ui-ux-designing-training-course-in-chennai.php">Best UI/UX Design Training in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/manual-testing-training-course-in-chennai.php">Software Testing Classes in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/digital-marketing-training-course-in-chennai.php">Digital Marketing Training Institute in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/adobe-experience-manager-training-course-in-chennai.php">AEM Online Training Course</a>
<a href="http://www.forerunsoftwaresolutions.com/adobe-experience-manager-training-course-in-chennai.php">CQ5 Development Course in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/laravel-training-course-in-chennai.php">Laravel Online Course In Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/laravel-training-course-in-chennai.php">Laravel Php Framework Training Course In Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/android-training-course-in-chennai.php">Android online Training in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/ios-training-course-in-chennai.php">IOS Online Training in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/ios-training-course-in-chennai.php">Best IOS Training Institutes in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/react-native-training-course-in-chennai.php">React Native Online Training in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/react-native-training-course-in-chennai.php">Best React Native Training Institutes in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/ui-ux-designing-training-course-in-chennai.php">Web Designing Course in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/ui-ux-designing-training-course-in-chennai.php">Best UI/UX Web Development Courses in chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/manual-testing-training-course-in-chennai.php">Manual Testing Training in Chennai</a>
<a href="http://www.forerunsoftwaresolutions.com/react-native-training-course-in-chennai.ph...

Read more...

Revision history for this message
In , varmaseema459 (varmaseema459-linux-kernel-bugs) wrote :

Thanks for sharing your wealthy information. This is one of the excellent posts. It is great to have the opportunity to read a good quality article with useful information.
https://technologycounter.com/hr-software/india

Revision history for this message
In , Wilbur310 (wilbur310-linux-kernel-bugs) wrote :
Download full text (11.6 KiB)

https://www.alelly.net/
https://www.alelly.net/collections/dresses/products/layered-ruffled-open-back-puff-sleeve-swiss-dot-mini-dress-1
https://www.alelly.net/collections/dresses/products/leopard-printed-o-neck-short-sleeve-dress-11
https://www.alelly.net/collections/dresses/products/square-neck-puff-sleeve-babydoll-style-short-dress-12
https://www.alelly.net/collections/dresses/products/spaghetti-straps-jacquard-ruffle-mini-dress-4
https://www.alelly.net/collections/dresses/products/loose-fit-ruffled-color-block-dress-3
https://www.alelly.net/collections/dresses/products/swiss-dot-long-sleeve-mini-dress-with-waist-tie-2
https://www.alelly.net/collections/dresses/products/spaghetti-straps-v-neck-smocked-swing-floral-dress-3
https://www.alelly.net/collections/dresses/products/v-neck-lace-shoulder-sleeveless-mini-dress-3
https://www.alelly.net/collections/dresses/products/brief-comfy-pleated-long-sleeve-white-mini-dress
https://www.alelly.net/collections/dresses/products/cold-shoulder-twist-knit-mini-dress-1
https://www.alelly.net/collections/dresses/products/v-neck-batwing-sleeve-maxi-dress
https://www.alelly.net/collections/dresses/products/v-neck-split-sleeve-sequin-dress-with-slit
https://www.alelly.net/collections/dresses/products/off-shoulder-lace-bodice-high-waist-maxi-skirt-evening-dress
https://www.alelly.net/collections/dresses/products/floral-square-neck-ruffles-backless-a-line-short-dress
https://www.alelly.net/collections/dresses/products/spaghetti-straps-silk-like-ruched-mini-dress
https://www.alelly.net/collections/dresses/products/ruffled-off-shoulder-floral-dress
https://www.alelly.net/collections/dresses/products/halloween-pumpkin-letter-print-striped-raglan-long-sleeve-mini-dress
https://www.alelly.net/collections/dresses/products/crew-neck-twist-hollow-out-t-shirt-mini-dress-with-ruched-detail
https://www.alelly.net/collections/dresses/products/asymmetric-cold-shoulder-denim-dress
https://www.alelly.net/collections/dresses/products/velvet-puff-sleeve-mini-dress
https://www.alelly.net/collections/dresses/products/leopard-print-spaghetti-strap-bodycon-mini-dress
https://www.alelly.net/collections/dresses/products/wrap-v-neck-long-sleeve-ruched-velvet-mini-dress
https://www.alelly.net/collections/dresses/products/pocket-lace-up-drawstring-hooded-mini-dress
https://www.alelly.net/collections/dresses/products/plaid-striped-splicing-raglan-sleeve-mini-dress
https://www.alelly.net/collections/dresses/products/layered-ruffled-open-back-puff-sleeve-swiss-dot-mini-dress-1
https://www.alelly.net/collections/dresses/products/leopard-printed-o-neck-short-sleeve-dress-11
https://www.alelly.net/collections/dresses/products/square-neck-puff-sleeve-babydoll-style-short-dress-12
https://www.alelly.net/collections/dresses/products/spaghetti-straps-jacquard-ruffle-mini-dress-4
https://www.alelly.net/collections/dresses/products/loose-fit-ruffled-color-block-dress-3
https://www.alelly.net/collections/dresses/products/swiss-dot-long-sleeve-mini-dress-with-waist-tie-2
https://www.alelly.net/collections/dresses/products/spaghetti-straps-v-neck-smocked-swing-floral-dress-3
https://www.alelly.net/collections/dresses/products/v-neck-lace-shoulder-sle...

Revision history for this message
In , Wilbur310 (wilbur310-linux-kernel-bugs) wrote :
Download full text (25.0 KiB)

https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08DD658R3
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08DD5HTL6
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08DD5J6QW
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08DD4JM3G
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4SZYP1
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4VCVKG
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4VGNVP
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4T886X
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4VXHVH
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4VJ4X8
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4VNWZM
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4TTDCM
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B0989GD7LN
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B0989BWS5G
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B09895GGQ3
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B0989CP2JW
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B099DZ28NL
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B099DVRCJV
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B099DVF4M3
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B099DX8FX9
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07TZ8B4QH
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07TVXPMDF
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07TVXPMDH
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07TZ89NYS
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07TZ89NYS
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07WY58VXN
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07WSYP624
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07WTZ5BZB
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07WNZ1QBS
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B07WNZ1QBS
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4TYZ8X
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4VM3XQ
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4VFQM4
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4V7ZPQ
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4TQ7CN
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4W5B4G
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4VN4PS
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08B4V2LBB
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08DD58ZJV
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08DD63Q1R
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08DD4MJT8
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B08DC71VXH
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B099DWY1HW
https://www.amazon.com/Alelly-Womens-Summer-Ruffle-Floral/dp/B099DYMKZ7
https://www.amazon.com/Alelly-Wo...

Displaying first 40 and last 40 comments. View all 833 comments or add a comment.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.