Horizon post 302 a hard reboot instance action. Instance does nothing

[Wendy Mitchell]

Brief Description
----------------
Horizon post 302 a hard reboot instance action. Instance does nothing

Severity
--------
Normal

steps:
Launch instance (tried on multiple different labs) eg. from image (or volume)
In horizon, attempt to Hard reboot the instance from the action menu (as tenant or admin)

Result
302 Post request is issued but I am not seeing evidence in the console that the instance actually reboots.

2018-12-12 20:56:08,736 [INFO] horizon.operation_log: [admin cb44b531905b4111a983812d53ff8453] [admin 5f37bd855658413abf42e1b916f56ece] [POST /auth/login/ 302] parameters:[{"fake_email": "", "username": "admin", "fake_password": "********", "region": "http://controller:5000/v3", "next": "/", "csrfmiddlewaretoken": "ET8XXE28SEb15BczBx5pQ8k5b3fhUeyv", "password": "********"}] message:[None]
2018-12-12 21:04:32,707 [INFO] horizon.tables.actions: Hard Rebooted Instance: "stress_ng"
2018-12-12 21:04:32,709 [INFO] horizon.operation_log: [admin cb44b531905b4111a983812d53ff8453] [admin 5f37bd855658413abf42e1b916f56ece] [POST /admin/instances/ 302] parameters:[{"instances__filter_admin_instances__q": "", "action": "instances__reboot__215a3ba2-03fb-4001-9892-7e712658d586", "csrfmiddlewaretoken": "K4Svppi3MZnr3GXVOysy9ItKqlpugMS1", "instances__filter_admin_instances__q_field": "project"}] message:[success: Hard Rebooted Instance: stress_ng]

Expected Behavior
------------------
Expected instance to reboot

Actual Behavior
----------------
instance does not reboot (nothing happens)

Reproducibility
---------------
100%

System Configuration
--------------------
any

Branch/Pull Time/Commit
-----------------------
2018-12-02_20-18-00 (also 2018-12-07_20-18-00)

Timestamp/Logs
--------------

2018-12-12 20:56:08,736 [INFO] horizon.operation_log: [admin cb44b531905b4111a983812d53ff8453] [admin 5f37bd855658413abf42e1b916f56ece] [POST /auth/login/ 302] parameters:[{"fake_email": "", "username": "admin", "fake_password": "********", "region": "http://controller:5000/v3", "next": "/", "csrfmiddlewaretoken": "ET8XXE28SEb15BczBx5pQ8k5b3fhUeyv", "password": "********"}] message:[None]
2018-12-12 21:04:32,707 [INFO] horizon.tables.actions: Hard Rebooted Instance: "stress_ng"
2018-12-12 21:04:32,709 [INFO] horizon.operation_log: [admin cb44b531905b4111a983812d53ff8453] [admin 5f37bd855658413abf42e1b916f56ece] [POST /admin/instances/ 302] parameters:[{"instances__filter_admin_instances__q": "", "action": "instances__reboot__215a3ba2-03fb-4001-9892-7e712658d586", "csrfmiddlewaretoken": "K4Svppi3MZnr3GXVOysy9ItKqlpugMS1", "instances__filter_admin_instances__q_field": "project"}] message:[success: Hard Rebooted Instance: stress_ng]

[Ghada Khalil (gkhalil)]

Marking as release gating; this is expected to work.
Unsure if this was previously tested on StarlingX or not.

[Bruce Jones (brucej)]

Cindy, please assign this issue to Yan.

[yong hu (yhu6)]

The issue was reproduced on Horizon.
As a reference I tried "soft reboot"and it worked well.
According to horizon and nova-conductor.log, the request(req-ba2f0c06-2be7-4b9b-8cc3-392304a92eb0) was created and passed to NOVA already.

## hard reboot request received in Nova-conductor
2019-01-14 19:39:13.552 23771 INFO nova.objects.instance_action [req-ba2f0c06-2be7-4b9b-8cc3-392304a92eb0 58318155edc742d888f7ee26affd87a0 f8ad5e7067c24148bae8c084149048fb - default default] compute__do_build_and_run_instance instance 735519d4-ab42-4f66-900e-2428de556b06

## Soft reboot request received in Nova-conductor
2019-01-15 18:28:01.341 23771 INFO nova.objects.instance_action [req-5497b285-88f7-42c6-a8ce-3bc442ca704d 58318155edc742d888f7ee26affd87a0 f8ad5e7067c24148bae8c084149048fb - default default] compute_reboot_instance instance 735519d4-ab42-4f66-900e-2428de556b06

Next step is to investigate why Nova wouldn't proceed this request.

[yong hu (yhu6)]

It turns out Horizon dashboard didn't correctly pass HARD_REBOOT to Nova.
Here is the code where this issue comes out:

@profiler.trace
def server_reboot(request, instance_id, soft_reboot=False):
    hardness = nova_servers.REBOOT_HARD
    if soft_reboot:
        hardness = nova_servers.REBOOT_SOFT
        novaclient(request).servers.reboot(instance_id, hardness)

If it is not soft_reboot, novaclient won't be called.
BTW: there is no such an issue in upstream Horizon project: https://github.com/openstack/horizon/blob/master/openstack_dashboard/api/nova.py

Will file a patch to fix this issue.

[yong hu (yhu6)]

https://github.com/starlingx-staging/stx-horizon/pull/20