Instance creation sometimes fails after host aggregate deletion

Bug #1605804 reported by Pooja Ghumre on 2016-07-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Low
Markus Zoeller (markus_z)
Mitaka
Low
Roman Podoliaka

Bug Description

Instance creation starts failing if nova scheduler gets in an inconsistent state wrt host aggregates. If remove_host_from_aggregate operation is invoked for multiple hosts in quick succession, followed by aggregate deletion, the nova scheduler host_manager maps (host_aggregates_map and aggs_by_id) get out of sync, as there are some stale references left behind in the host_aggregates_map for an aggregate that is deleted from the aggs_by_id map.
This is because it cleans up state based on aggregate.hosts which is empty when aggregate is deleted, but the prior aggregate updates to remove individual hosts could have incorrect list of hosts added to the host_aggregates_map.

Instance creation fails with below error once scheduler gets in this state:
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher [req-7f29701b-0272-444c-8650-a1035777e642 d2c755daa21e451e86c1d2b5be705aa2 0546d7f9c747456aa0ffb306cfe5627d - - -] Exception during message handling: 1
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher return func(*args, **kwargs)
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/manager.py", line 84, in select_destinations
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher filter_properties)
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 72, in select_destinations
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher filter_properties)
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 164, in _schedule
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher hosts = self._get_all_host_states(elevated)
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 222, in _get_all_host_states
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher return self.host_manager.get_all_host_states(context)
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher File "/opt/pf9/nova/lib/python2.7/site-packages/nova/scheduler/host_manager.py", line 585, in get_all_host_states
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher host_state.host]]
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher KeyError: 1
2016-07-21 18:20:16.780 15692 ERROR oslo_messaging.rpc.dispatcher
2016-07-21 18:20:16.784 15692 ERROR oslo_messaging._drivers.common [req-7f29701b-0272-444c-8650-a1035777e642 d2c755daa21e451e86c1d2b5be705aa2 0546d7f9c747456aa0ffb306cfe5627d - - -] Returning exception 1 to caller

Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :
Changed in nova:
status: New → Confirmed
importance: Undecided → Low
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :

We should try to reproduce this with a functional regression test.

tags: added: compute
tags: added: scheduler
Revision history for this message
Markus Zoeller (markus_z) (mzoeller) wrote :
Changed in nova:
status: Confirmed → In Progress
assignee: nobody → Markus Zoeller (markus_z) (mzoeller)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/352344
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f0dd4d6bdd286ea155cf55eb62662993577d8892
Submitter: Jenkins
Branch: master

commit f0dd4d6bdd286ea155cf55eb62662993577d8892
Author: Markus Zoeller <email address hidden>
Date: Mon Aug 8 12:46:43 2016 +0200

    Fix corrupt "host_aggregates_map" in host_manager

    A host can be in multiple host-aggregates at the same time. When a
    host gets removed from an aggregate in thread A and this aggregate
    gets deleted in thread B, there can be a race-condition where the
    mapping data in the host_manager can get out of sync for a moment.

    This change simulates this condition in a unit test and fixes the bug
    by iterating over the mapping itself instead of the out-of-date list
    "aggregates.hosts".

    Closes-Bug: 1605804
    Change-Id: I59861f03f0c681f7118782fb017af377e07552aa

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.0.0rc1

This issue was fixed in the openstack/nova 14.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/381860

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (stable/mitaka)

Change abandoned by Roman Podoliaka (<email address hidden>) on branch: stable/mitaka
Review: https://review.openstack.org/381860
Reason: too late for mitaka

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.0.0rc1

This issue was fixed in the openstack/nova 14.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers