VMware: running a redundant nova compute deletes running instances

Bug #1419785 reported by Gary Kotton on 2015-02-09
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Gary Kotton
Juno
Undecided
Unassigned

Bug Description

When running more than one nova compute configured for the same cluster, rebooting one of the computes will delete all running instances.

Fix proposed to branch: master
Review: https://review.openstack.org/154029

Changed in nova:
assignee: nobody → Gary Kotton (garyk)
status: New → In Progress
Gary Kotton (garyk) on 2015-02-09
Changed in nova:
importance: Undecided → Critical
Matthew Booth (mbooth-9) wrote :

Some context: this happens because _destroy_evacuated_instances in compute.manager does (lightly edited for clarity):

        local_instances = self._get_instances_on_driver(context, filters)
        for instance in local_instances:
            if instance.host != self.host:
                ...DESTROY...

The only instances which will be destroyed are the ones for which instance.host != self.host.

The meaning of self.host in this context appears to be 'hypervisor'. However, self.host is also a service endpoint. Historically there was a 1:1 relationship between these 2, but there are now a couple of drivers where this no longer makes sense.

I think the correct fix for this would be something like adding driver.get_hypervisor_id() which returns a driver-specific identifier for the hypervisor location. Instance.host would then be set to this value. HA nova instances would then ensure that this returned the same value for all Novas managing the same hypervisor.

However, that's a spec and a bunch of work, and this is a critical issue.

Note that there is no problem in the above code if the active and standby node have the same value of self.host. The immediate workaround would seem to be to configure the active and standby nodes accordingly. This would presumably assume simultaneous failover of dns/ip.

For this specific issue, I would prefer to see a solution which is able to detect this situation and refuse to start Nova.

Fix proposed to branch: master
Review: https://review.openstack.org/154907

Changed in nova:
assignee: Gary Kotton (garyk) → Matthew Booth (mbooth-9)
Gary Kotton (garyk) wrote :

The fix that Matt suggest is not viable in my opinoin. It does not support HA at all which kind of defeats the purpose

Matthew Booth (mbooth-9) wrote :

Supporting HA isn't in the scope of this Critical bug. Nova doesn't currently support HA, and adding it will require a spec and a significant amount of work. This bug will fix the bug as described here.

That said, I believe HA will still be supportable if all HA nodes have the same hostname.

tags: added: vmware
Changed in nova:
assignee: Matthew Booth (mbooth-9) → Gary Kotton (garyk)
Michael Still (mikal) wrote :

Cannot be critical, as single driver.

Changed in nova:
importance: Critical → High

Change abandoned by Matthew Booth (<email address hidden>) on branch: master
Review: https://review.openstack.org/154907
Reason: I think this can be better done in the DB, which is cleaner and applies to all drivers.

Fix proposed to branch: master
Review: https://review.openstack.org/158269

Changed in nova:
assignee: Gary Kotton (garyk) → Matthew Booth (mbooth-9)
Matthew Booth (mbooth-9) wrote :

Michael, I believe this bugs meets the definition of critical here:

https://wiki.openstack.org/wiki/BugTriage#Task_2:_Prioritize_confirmed_bugs_.28bug_supervisors.29

because it results in data loss. Severe data loss, in fact. It also affects both the VMware and Ironic drivers.

Changed in nova:
importance: High → Critical
Joe Gordon (jogo) wrote :

critical is for things that impact all users

Changed in nova:
importance: Critical → High

Fix proposed to branch: master
Review: https://review.openstack.org/159890

Changed in nova:
assignee: Matthew Booth (mbooth-9) → Dan Smith (danms)
Changed in nova:
assignee: Dan Smith (danms) → Gary Kotton (garyk)
Changed in nova:
assignee: Gary Kotton (garyk) → Matthew Booth (mbooth-9)
Changed in nova:
assignee: Matthew Booth (mbooth-9) → Dan Smith (danms)
Changed in nova:
assignee: Dan Smith (danms) → Gary Kotton (garyk)

Reviewed: https://review.openstack.org/159890
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=922148ac45c5a70da8969815b4f47e3c758d6974
Submitter: Jenkins
Branch: master

commit 922148ac45c5a70da8969815b4f47e3c758d6974
Author: Dan Smith <email address hidden>
Date: Fri Feb 27 07:30:10 2015 -0800

    Allow disabling the evacuate cleanup mechanism in compute manager

    This mechanism attempts to destroy any locally-running instances on
    startup if instance.host != self.host. The assumption is that the
    instance has been evacuated and is safely running elsewhere. This is
    a dangerous assumption to make, so this patch adds a configuration
    variable to disable this behavior if it's not desired.

    Note that disabling it may have implications for the case where
    instances *were* evacuated, given potential shared resources.
    To counter that problem, this patch also makes _init_instance()
    skip initialization of the instance if it appears to be owned
    by another host, logging a prominent warning in that case.

    As a result, if you have destroy_after_evacuate=False and you start
    a nova compute with an incorrect hostname, or run it twice from
    another host, then the worst that will happen is you get log
    warnings about the instances on the host being ignored. This should
    be an indication that something is wrong, but still allow for
    fixing it without any loss. If the configuration option is disabled
    and a legitimate evacuation does occur, simply enabling it and then
    restarting the compute service will cause the cleanup to occur.

    This is added to the workarounds config group because it is really
    only relevant while evacuate is fundamentally broken in this way.
    It needs to be refactored to be more robust, and once that is done,
    this should be able to go away.

    DocImpact: New configuration option, and peril warning
    Partial-Bug: #1419785
    Change-Id: Ib9a3c72c096822dd5c65c905117ae14994c73e99

Change abandoned by Matthew Booth (<email address hidden>) on branch: master
Review: https://review.openstack.org/158269
Reason: Fucked if I know why.

Change abandoned by garyk (<email address hidden>) on branch: master
Review: https://review.openstack.org/154029
Reason: Need to discuss this

Reviewed: https://review.openstack.org/174779
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6f1f9dbc211356a3d0e2d46d3a984d7ceee79ca6
Submitter: Jenkins
Branch: stable/juno

commit 6f1f9dbc211356a3d0e2d46d3a984d7ceee79ca6
Author: Tony Breeds <email address hidden>
Date: Tue Jan 27 11:17:54 2015 -0800

    Allow disabling the evacuate cleanup mechanism in compute manager

    This mechanism attempts to destroy any locally-running instances on
    startup if instance.host != self.host. The assumption is that the
    instance has been evacuated and is safely running elsewhere. This is
    a dangerous assumption to make, so this patch adds a configuration
    variable to disable this behavior if it's not desired.

    Note that disabling it may have implications for the case where
    instances *were* evacuated, given potential shared resources.
    To counter that problem, this patch also makes _init_instance()
    skip initialization of the instance if it appears to be owned
    by another host, logging a prominent warning in that case.

    As a result, if you have destroy_after_evacuate=False and you start
    a nova compute with an incorrect hostname, or run it twice from
    another host, then the worst that will happen is you get log
    warnings about the instances on the host being ignored. This should
    be an indication that something is wrong, but still allow for
    fixing it without any loss. If the configuration option is disabled
    and a legitimate evacuation does occur, simply enabling it and then
    restarting the compute service will cause the cleanup to occur.

    This is added to the workarounds config group because it is really
    only relevant while evacuate is fundamentally broken in this way.
    It needs to be refactored to be more robust, and once that is done,
    this should be able to go away.

    Conflicts:
            nova/compute/manager.py
            nova/tests/unit/compute/test_compute.py
            nova/tests/unit/compute/test_compute_mgr.py
            nova/utils.py

    NOTE: In nova/utils.py a new section has been introduced but
    only the option addessed by this backport has been included.

    DocImpact: New configuration option, and peril warning
    Partial-Bug: #1419785
    (cherry picked from commit 922148ac45c5a70da8969815b4f47e3c758d6974)

    -- squashed with commit --

    Create a 'workarounds' config group.

    This group is for very specific reasons.

    If you're:
    - Working around an issue in a system tool (e.g. libvirt or qemu) where the fix
      is in flight/discussed in that community.
    - The tool can be/is fixed in some distributions and rather than patch the code
      those distributions can trivially set a config option to get the "correct"
      behavior.
    This is a good place for your workaround.

    (cherry picked from commit b1689b58409ab97ef64b8cec2ba3773aacca7ac5)

    --

    Change-Id: Ib9a3c72c096822dd5c65c905117ae14994c73e99

tags: added: in-stable-juno
Dan Smith (danms) wrote :

The actual fix for this is to make nova-compute not attempt to delete instances unless they were actually evacuated. That fix was committed here, so this should be fixed now:

https://review.openstack.org/#/c/183354

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2015-09-24
Changed in nova:
milestone: none → liberty-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2015-10-15
Changed in nova:
milestone: liberty-rc1 → 12.0.0
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers