Scheduler AZ filter should use cached values of host to AZ mapping

Bug #1189557 reported by Phil Day
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Phil Day

Bug Description

If the scheduler is creating multiple instances in the same request it will re-run each filter for each host for each instance.

In the case of the AZ filter, this currently includes a DB look up to get the AZ value from the aggregate - although this value isn't going to change on successive runs.

Where there are a lot of hosts this can take several seconds for each run of the filter. When repeating this say 100 times (to create 100 instances) the total time taken by the scheduler can exceed the service timeout interval, and so hosts start getting dropped by the compute filter because it looks as if the last service update (cached by the scheduler at the start of the run has now expired).

There seem to be two possible solutions here:
i) Change the AZ filter to use a cached value (Would be simple as the availability zone class already has caching)

ii) Change the filter mechanism so that filters can be defined so that they need only to be run once for each scheduler request

 The second approach seems more general

Tags: scheduler
Phil Day (philip-day)
Changed in nova:
assignee: nobody → Phil Day (philip-day)
Revision history for this message
Joe Gordon (jogo) wrote :

Phil,

You can implement a caching mechanism in the filters by using https://review.openstack.org/#/c/29343/12/nova/scheduler/filters/volume_affinity_filter.py as a guide. Filters get initialized for each RPC request coming into the scheduler, so the above patch uses __init__ to help with caching. Furthermore this is a common problem in many filters, so it may be worth it to abstract out the caching code so other filters can re-use it.

Revision history for this message
Phil Day (philip-day) wrote : Re: [Bug 1189557] Re: Scheduler AZ filter should use cached values of host to AZ mapping

Thanks Joe I'll take a look. I'm also thinking about a general property in the filter base class to indicate that a filer only needs to be run once per request, as I think there a number that could use that

Sent from Samsung Mobile

-------- Original message --------
Subject: [Bug 1189557] Re: Scheduler AZ filter should use cached values of host to AZ mapping
From: Joe Gordon <email address hidden>
To: "Day, Phil" <email address hidden>
CC:

Phil,

You can implement a caching mechanism in the filters by using
https://review.openstack.org/#/c/29343/12/nova/scheduler/filters/volume_affinity_filter.py
as a guide. Filters get initialized for each RPC request coming into
the scheduler, so the above patch uses __init__ to help with caching.
Furthermore this is a common problem in many filters, so it may be worth
it to abstract out the caching code so other filters can re-use it.

--
You received this bug notification because you are subscribed to the bug
report.
https://bugs.launchpad.net/bugs/1189557

Title:
  Scheduler AZ filter should use cached values of host to AZ mapping

Status in OpenStack Compute (Nova):
  New

Bug description:
  If the scheduler is creating multiple instances in the same request it
  will re-run each filter for each host for each instance.

  In the case of the AZ filter, this currently includes a DB look up to
  get the AZ value from the aggregate - although this value isn't going
  to change on successive runs.

  Where there are a lot of hosts this can take several seconds for each
  run of the filter. When repeating this say 100 times (to create 100
  instances) the total time taken by the scheduler can exceed the
  service timeout interval, and so hosts start getting dropped by the
  compute filter because it looks as if the last service update (cached
  by the scheduler at the start of the run has now expired).

  There seem to be two possible solutions here:
  i) Change the AZ filter to use a cached value (Would be simple as the availability zone class already has caching)

  ii) Change the filter mechanism so that filters can be defined so that
  they need only to be run once for each scheduler request

   The second approach seems more general

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1189557/+subscriptions

melanie witt (melwitt)
tags: added: scheduler
Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/33720
Committed: http://github.com/openstack/nova/commit/4a03071db23196c4ac1ec0d8ea163c46887d3b08
Submitter: Jenkins
Branch: master

commit 4a03071db23196c4ac1ec0d8ea163c46887d3b08
Author: Phil Day <email address hidden>
Date: Wed Jun 19 21:09:42 2013 +0100

    Allow filters to only run once per request if their data is static

    Currently the filter_all() method of each scheduler filter is run
    once for each instance in a request, but for many filters the data
    doesn't change during a request.

    For example the data used by the AZ filter is pretty static, and
    doesn't filter any more hosts on successive runs. However it is
    fairly expensive to run on a large system.

    Similarly the ServiceGroup information used by the compute filter
    is cached as part of the host status, and doesn't need to be
    evaluated more than once per request.

    Only filters that are based on data that may change within a
    request, for example resource consumption, need to be evaluated
    for each instance.

    This blueprint introduces a new attribute that allows filters to
    declare that they only need to be run once per request. The default
    behaviour is left so that a filter is evaluated once for each
    instance in a request.

    The following existing filters are changed to only evaluate once
    per request:

      AggregateInstanceExtraSpecsFilter
      AggregateMultiTenancyIsolation
      AggregateTypeAffinityFilter
      AllHostsFilter
      AvailabilityZoneFilter
      ComputeCapabilitiesFilter
      ComputeFilter
      DifferentHostFilter
      GroupAntiAffinityFilter
      ImagePropertiesFilter
      IsolatedHostsFilter
      SameHostFilter
      SimpleCIDRAffinityFilter

    In addition the function that does the check is defined in the
    filter base class in a way that a filter that wants to run
    (for example) once for every 10 instances in a request could
    overload the function and implement its own behaviour.

    Implements: bp once-per-request-filters
    Fixes: bug 1189557

    Change-Id: Ia668f16414da86441323c58b1bbef5f88c81b90c

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → havana-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: havana-2 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.