Mgmt Reboot Should Not Be Rejected If In CRASHED State or Nova in SHUTDOWN State

[Auston McReynolds]

If the DB status is CRASHED, a MGMT Reboot should work; however, it does not at the moment.

It's clear that the intent is that it should, from:

    @property
    def action_is_allowed(self):
        allowed_statuses = [
            ServiceStatuses.RUNNING._code,
            ServiceStatuses.SHUTDOWN._code,
            ServiceStatuses.CRASHED._code,
            ServiceStatuses.BLOCKED._code,
        ]
        return self._code in allowed_statuses

but, unfortunately,

    def reboot(self):
        try:
            LOG.debug("Stopping datastore on instance %s." % self.id)
            self.guest.stop_db()

reboot as it's first step calls stop_db(), which:

    def stop_db(self, update_db=False, do_not_start_on_reboot=False):
        LOG.info(_("Stopping MySQL."))
        if do_not_start_on_reboot:
            self._disable_mysql_on_boot()
        try:
            mysql_service = operating_system.service_discovery(
                MYSQL_SERVICE_CANDIDATES)
            utils.execute_with_timeout(mysql_service['cmd_stop'], shell=True)
        except KeyError:
            LOG.exception(_("Error stopping MySQL."))
            raise RuntimeError("Service is not discovered.")
        if not self.status.wait_for_real_status_to_change_to(
                rd_instance.ServiceStatuses.SHUTDOWN,
                self.state_change_wait_time, update_db):
            LOG.error(_("Could not stop MySQL."))
            self.status.end_install_or_restart()
            raise RuntimeError("Could not stop MySQL!")

will raise a RuntimeError of "Could not stop MySQL!" because the status remains CRASHED vs. SHUTDOWN, causing the reboot operation to immediately bail out before rebooting the instance.

There are a couple of ways to fix this, some better than others.

---

Update 09/30: found another variation of this that should be fixed at the same time.

it's possible that mgmt-show shows:

| guest_status | running |
| id | 833914a8-202d-4826-9961-efe2ed60cf68 |
| ip | 10.64.212.69 |
| service_status | ACTIVE |
| status | ACTIVE |

but the nova instance according to nova shows as:

OS-EXT-STS:power_state = 4
OS-EXT-STS:task_state = -
OS-EXT-STS:vm_state = stopped
status = SHUTOFF

when executing a mgmt-reboot,

$ supernova lvs-admin mgmt-reboot 833914a8-202d-4826-9961-efe2ed60cf68
[SUPERNOVA] Running trove against lvs-admin...
ERROR: Instance 833914a8-202d-4826-9961-efe2ed60cf68 is not currently available for an action to be performed (status was SHUTOFF). (HTTP 422)

this is problematic because it requires a "nova reboot" and then a "trove mgmt-reboot" to correctly re-attach the volume and start the database.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/125849

[Ramashri Umale (ramashri)]

discussed with amcrn:
Mgmt Reboot will still be rejected when nova compute instance is in shutdown state. The reasoning being

1. nova reboot is not allowed for instances in shutoff state
2. nova compute shutdown state may be an intermediate state for a nova migrate or reboot operation. If so causing a nova reboot from trove would seem very risk prone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/125849
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=b23eddeb1743e0118e04362068c53789d4383ac0
Submitter: Jenkins
Branch: master

commit b23eddeb1743e0118e04362068c53789d4383ac0
Author: rumale <email address hidden>
Date: Thu Oct 2 16:27:49 2014 -0700

    Mgmt Reboot allowed if datastore in crashed state

    Change-Id: I256c74bc3f9cadd1c4dce4abd4ddc51cbba71231
    Closes-Bug: 1365185