Trove Resize doesn't wait for instance to become ACTIVE on nova

[Saurabh Surana]

Trove after confirming the resize on nova, doesn't wait for the instance to become ACTIVE, and in certain cases, befor nova makes that instance ACTIVE, trove marks it as active in its database(as trove resize function finishes and even the guest agent has updated the status in the DB to be active/running). But since the status on nova is still VERIFY_RESIZE, some trove-apis like reboot, which involves a nova action, starts failing as the instance stats on nova is not ACTIVE.

The error received by the client :
{"unprocessableEntity": {"message": "Instance is not currently available for an action to be performed (status was VERIFY_RESIZE).", "code": 422}}

This is hard to reproduce as it involves some behavior of nova, where it takes little longer to mark instance as ACTIVE after issuing resize-confirm.

The fix would be to wait for the instance to become ACTIVE in nova before sending the message to the guest agent.

[Haomai Wang (haomai)]

I guess:
1. Trove cast a resize request and will wait for instance becoming ACTIVE
2. Trove send a refresh instance status request
3. Nova reply instance status before handle reboot request, Trove will get instance status which is ACTIVE
4. Trove happily thinks resize action is finishied.

Maybe more logs can help verify the truth

[Michael Basnight (hubcap)]

Can you confirm this bug still exists?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/60346

[Sushil Kumar (sushil-kumar2)]

trove should not update the status prior to nova activation.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/60346
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=25303f1f8df0e91e835360869a61cbf3eca30f60
Submitter: Jenkins
Branch: master

commit 25303f1f8df0e91e835360869a61cbf3eca30f60
Author: Paul Marshall <email address hidden>
Date: Thu Dec 5 14:42:10 2013 -0600

    Acknowledge Nova VERIFY_RESIZE as RESIZE state

    After a resize, Nova changes instances to the VERIFY_RESIZE state,
    which must be confirmed before they return to ACTIVE. However, as
    far as Trove is concerned instances in VERIFY_RESIZE should still
    report RESIZE until they're actually done. Additionally, instance
    status unit tests are added to verify that the correct statuses
    are reported.

    Change-Id: I9c1824f8fe420025f329209c41c7aac1d8d7921c
    Closes-Bug: #1217985