[astute] OS install failure blocks environment

[Damia Pastor]

Scenario:

While deploying multiple nodes using Ubuntu, one of the nodes has a hardware failure and restarts. While Fuel tries to reinstall again, either stop nor reset will work, leaving the process in a loop.

Steps to reproduce:

- Deploy multiple nodes
- During OS installation (tested with Ubuntu), hard reset the node simulating a hardware failure
- Either by using GUI or CLI, stop the process.

Expected behaviour:

- Deployment is stopped and Fuel goes back to the pre-deploy status for the environment

Actual behaviour:

- The deployment blocks itself, repeating in a loop the OS installation. There is no cancel option.

Notes:

- We also tried a "reset" from fuel cli and deleting the environment.

[Damia Pastor (magradallegir)]

Hi Aleksey,

I think that this was caused by an idle user syndrome: The user gets nervous as the task takes too long, so we start trying things:

- Cancel, Reset, Delete

I will retest if I can still reproduce it, if not, I will open a new ticket to block Reset and delete actions when cancel is ongoing.

Thanks and sorry for the inconvenience.

[Stanislaw Bogatkin (sbogatkin)]

I can't reproduce that.

Tried on CentOS HA with 3 controllers and force reboot one of them, then stop deployment - all stopped ok.
And on Ubuntu simple with 5 nodes and force reboot two of them, then stop deployment - all stopped ok.

[Vladimir Sharshov (vsharshov)]

This behavior really very strange, because in case of Stop Deployment we kill main process (prevent conflicting commands), then try to access to all cluster's nodes via ssh and erase them. This step can take 5 minutes by default (60 seconds, 5 retries) and after it we remove nodes from Cobbler and after it send nodes to reboot via ssh.

After the above i think that idea from Damià Pastor is closest to the truth.

Unfortunately without logs i could not say more specifically.

[Damia Pastor (magradallegir)]

No problem Vladimir,

I will try to work better my case and provide logs. I think there are two points of interest:

a) What happens if Fuel can't reach one of the nodes, scheduled for deployment.

b) Should we block any interaction with the cluster (delete, reset) once it is in stop process.

I will work further on these two scenarios and create a new bug if necessary. :)

[Matthew Mosesohn (raytrac3r)]

Is there an update on this bug?

[Damia Pastor (magradallegir)]

Hi Matthew,

I need to differentiate the two cases, and re-test them again:

a) Nodes are not coming back, user only does stop. In 1 of 3 it might happen the operation gets stuck (reseted after 24h).
b) Nodes are not coming back, user stops, then tries to delete the environment. Got stuck in 3 of 3 cases (reseted after 24h).

I will check if it is possible to kill the task through CLI as a feasible workaround, based on support tickets.

[Mike Scherbakov (mihgen)]

Vladimir,
any update on this bug?

[Oleksiy Molchanov (omolchanov)]

This bug was incomplete for more than 4 weeks. We cannot investigate it further so we are setting the status to Invalid. If you think it is not correct, please feel free to provide requested information and reopen the bug, and we will look into it further.