heat down instance create build bug

[Kevin Fox]

If Heat is not responding during a Trove instance create, the instance gets stuck in BUILD state forever and never recovers.

It also seems like there is no way any more to delete the instance in this state.

[Denis M. (dmakogon)]

i'd suggest to grep service catalog (from context) and see if there's an endpoint for heat, if not - raise an exception and makr instance as FAILED.

[Kevin Fox (kevpn)]

no, the endpoint is there. the heat api was not responding at the time of trove instance creation. I had to restart heat-api/engine before it became responsive again. But the same issue can happen, if there is a problem with network connectivity. Trove should not get stuck forever in BUILDING state and not allow you to fix it without database manipulations.

[Denis M. (dmakogon)]

In this case you have to validate that API is responsive, just call 'stack list' before going to create a new stack.

[Denis M. (dmakogon)]

But, in general, it seems like a deployment problem, not Trove, at all. You have to verify that heat API is reachable from Trove API host. So, i think this is not a bug.

[Kevin Fox (kevpn)]

So, if heat is down, trove should let an instance get stuck in BUILDING state forever and never let the user use or delete the instance but still dock their quota for a nonfunctional instance? That sounds like a bug to me. As a user, I dont expect to have to check on heat before launching a trove instance. I'd expect the instance to go into ERROR state on the failure, so I can delete it. That it doesn't is a bug.

[Denis M. (dmakogon)]

In production deployment you don't have only one Heat API service running (with respect to Load Ballancers). So i still stand on that this is not a Trove bug, it's any deployment mistake. Nothing else.

[Amrith Kumar (amrith)]

Kevin Fox, please update with additional information to help debug this or close this bug. Thanks!

[Kevin Fox (kevpn)]

Simple test case.
 * configure trove to use heat.
 * stop the heat api endpoint.
 * use trove to launch an instance.
 * wait a few seconds seconds.
 * start the heat api endpoint.

You now have a trove instance stuck in status=BUILD forever. At very least, it should error out at some point. Better would be for it to retry the heat stack-create multiple times before giving up.

I have yet to figure out a way to unstick it or delete the instance in trove without mucking around in the database.

[Nikhil Manchanda (slicknik)]

So we have an instance usage_timeout which basically configures the amount of time an instance can stay in BUILD before it is marked as ERROR (so that the user can go and clean it up.)

If this is not happening in the case when we use heat to provision an instance, that seems like a bug to me.

[Kevin Fox (kevpn)]

Yeah that would be great. It would also solve a similar problem I just ran into. I tried launching a slave mysql off of my existing working instance. The master went to BACKUP state, and then the process died since I don't have swift enabled on my test cloud. So I have two instances now, one stuck in BACKUP for the last 3 days, and a second one the slave, stuck in BUILD state. Both undeletable.

[Amrith Kumar (amrith)]

heat support (trove using heat to provision instances) is being eliminated.