AIO-SX Low-latency: Watchdog fires while installing openstack

[Brent Rowsell]

Brief Description
-----------------
I attempted to install openstack on an AIO-SX with the low-latency profile. Towards the end of the installation, the load average went to 400+ lost terminal response and eventually the watchdog fired and rebooted the system.

Severity
--------
Critical+

Steps to Reproduce
------------------
See above

Expected Behavior
------------------
System does not implode

Actual Behavior
----------------
System implodes

Reproducibility
---------------
100%

System Configuration
--------------------
AIO-SX low-latency

Branch/Pull Time/Commit
-----------------------
"2019-06-12 20:20:07 -0400"

Last Pass
---------
This worked on a June 4th load.

Timestamp/Logs
--------------
Will attach logs

Test Activity
-------------
Other

[Brent Rowsell (brent-rowsell)]

Issue including the high load average is not seen on a load built on June 4th.

[Cindy Xie (xxie1)]

@Alex, any progress that you can share for this critical bug?

[Alexander Kozyrev (akozyrev)]

we finally got some kernel stacks from a dead system and analyzing them now.
Looks like all the platform CPUs seem to be trying to acquire spinlocks.
Backtraces are attached.

[Alexander Kozyrev (akozyrev)]

back-porting this commit form kernel mainline should cure the issue:
https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=c0ad4aa4d8416a39ad262a2bd68b30acd951bf0e
testing of this patch is in progress, for details refer to this thread:
https://<email address hidden>/T/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/673028

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/673028
Committed: https://git.openstack.org/cgit/starlingx/integ/commit/?id=56a91fec13cd018f998a0bdf0beed753efd94df5
Submitter: Zuul
Branch: master

commit 56a91fec13cd018f998a0bdf0beed753efd94df5
Author: Alex Kozyrev <email address hidden>
Date: Fri Jul 26 13:58:17 2019 -0400

    Backport the fix for deadlock in CFS-bandwidth timer locking

    Low-latency profile of StarlingX is affected by a deadlock in
    CFS scheduler. spin_lock is used in IRQ handler there instead of
    spin_lock_irqsave. This leads to an attempt to lock the same
    spinlock twice and inevitable system freeze. Backporting c0ad4aa4d8
    commit from upstream kernel to cure the issue.

    Change-Id: I5416c0e0886f42d2bcec8e3e5da063e6af6916f8
    Closes-bug: 1832854
    Signed-off-by: Alex Kozyrev <email address hidden>