standby controller not going active after forced reboot of active controller

[Chris Friesen]

Brief Description
-----------------
In a two-node system, when forcing the active controller to go into an uncontrolled reboot (by crashing the kernel via sysrq) the standby controller sometimes takes a long time (multiple minutes) to go active.

On possible issue is that I think the system in question has direct-connected mgmt/infra links.

Severity
--------
Major

Steps to Reproduce
------------------
On active controller, as root, run:
echo 1 > /proc/sys/kernel/sysrq; echo c > /proc/sysrq-trigger

Expected Behavior
------------------
The standby controller should go active.

Actual Behavior
----------------
The standby controller stayed standby for multiple minutes.

Reproducibility
---------------
Intermittent but frequent

System Configuration
--------------------
Two node system, and I think the mgmt/infra links are direct connect

Branch/Pull Time/Commit
-----------------------
<email address hidden>"
BUILD_NUMBER="6"
BUILD_HOST="starlingx_mirror"
BUILD_DATE="2019-03-05 06:00:00 +0000"

Timestamp/Logs
--------------
Timestamp is somewhere around 19:46:56

[Chris Friesen (cbf123)]

controller-0 sm logs

[Chris Friesen (cbf123)]

controller-1 sm logs

[Ghada Khalil (gkhalil)]

Marking as release gating; issue related to ha

[Bin Qian (bqian20)]

The issue was not reproducible with newest build (May 14).

[Bin Qian (bqian20)]

Test on AIO-DX and AIO-DX/DC.