GroupAntiAffinityFilter scheduler hint no longer works

Bug #1296913 reported by David Patterson
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Russell Bryant

Bug Description

Passing a scheduler hint for the GroupAntiAffinityFilter no longer works:

nova boot --flavor m1.nano --image cirros-0.3.1-x86_64-uec --nic net-id=909e7fa9-b3af-4601-84c2-01145b1dea72 --hint group=foo server-foo

ERROR (NotFound): The resource could not be found. (HTTP 404) (Request-ID: req-21430f41-e6ca-46db-ab5c-890a1d1dbd01)

screen-n-api.log contains message:

Caught error: Instance group foo could not be found.

Tags: scheduler
Revision history for this message
Tracy Jones (tjones-i) wrote :

please attach logs

tags: added: scheduler
Changed in nova:
status: New → Incomplete
Revision history for this message
Dan Smith (danms) wrote :

This looks like a regression due to the recent merge of instance groups

Changed in nova:
milestone: none → icehouse-rc1
importance: Undecided → High
Revision history for this message
David Patterson (david-patterson) wrote :
Revision history for this message
Russell Bryant (russellb) wrote :

The problem here is that with instance groups, you *have* to create the great in advance. You have to create the group with the policy it's going to use before booting any instances into that group. We don't allow you to set policy on an existing group, because we have no way to apply that to existing instances.

Revision history for this message
Russell Bryant (russellb) wrote :

My suggestion is that we just accept this as the way it worked. Specifying a group=foo hint from the API actually had *no* defined meaning before. It may or may not have done anything, depending on how the scheduler was configured.

Now it has a very specific meaning. If you specify this hint, the instance created will get added to that group. Policy set on that group will be enforced when the scheduler goes to schedule the instance, without any changes needed to scheduler config.

Revision history for this message
Russell Bryant (russellb) wrote :

Actually, let me try to summarize the change better ...

In Havana:

If you configured the filter scheduler to include the GroupAntiAffintyFilter, you could boot with the group=foo scheduler hint. The group would get stored in the instance's system_metadata. Any other instances with that same group in their system metadata would be put into a list to be used by the filter for enforcing affinity.

In Icehouse:

You can enable both GroupAffinityFilter and GroupAntiAffinityFilter for the filter scheduler at the same time. You must define an instance group named 'foo' with a policy of 'anti-affinity'. Then, you can boot an instance with a scheduler hint of group=foo. That instance will get added to the group and the anti-affinity filter will automatically get used since it's in a group with that policy set.

This is certainly a change in behavior. The problem really is that we released incomplete group support without the API in Havana. Technically you could have used it, and the changes we've made break you if you used it the way it was.

Changed in nova:
status: Incomplete → Confirmed
Revision history for this message
Russell Bryant (russellb) wrote :

Proposed solution to be backwards compatible:

1) if group=foo is specified and group 'foo' does not exist, automatically create it. Set a special policy type of "scheduler" on the automatically created group.

2) Rename the group scheduler filters that work on the 'affinity' and 'anti-affinity' policies of instance groups.

3) Add new filters with the old names that apply affinity/anti-affinity for instances in groups with the policy of 'scheduler'. They will get applied if you configure the scheduler to use them.

That should result in allow both the new style behavior if you enable the new filters, as well as retain backwards compatibility with the old code / behavior.

Changed in nova:
assignee: nobody → Russell Bryant (russellb)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/83588

Changed in nova:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/83589

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/83590

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/83588
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c447d6c043256d84999e096256fa96a04466fc1e
Submitter: Jenkins
Branch: master

commit c447d6c043256d84999e096256fa96a04466fc1e
Author: Russell Bryant <email address hidden>
Date: Thu Mar 27 18:45:16 2014 +0000

    Add InstanceGroup.get_by_name()

    Legacy behavior of the 'group' scheduler hint was that the value was
    treated as a name for the group. To preserve the old behavior, the
    InstanceGroup object needs to be able to look up a group by name.

    Since we're so close to doing an RC, just use the db API methods
    available now but leave a note that this should be optimized later
    with a db.api query that looks up the group by name.

    Change-Id: I1b8c72b29a9a44a171aaa231aeb3533ccf41f7c2
    Partial-bug: #1296913

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/83589
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ea6d8403bdc3c1d3ede87a2ed1c8740420c32080
Submitter: Jenkins
Branch: master

commit ea6d8403bdc3c1d3ede87a2ed1c8740420c32080
Author: Russell Bryant <email address hidden>
Date: Thu Mar 27 17:54:20 2014 +0000

    Automatically create groups that do not exist

    If the scheduler hint 'group' is specified and the contents are not a
    UUID, automatically create the group. This is to maintain backwards
    compatibility with the old behavior of the 'group' scheduler hint.

    The policy on this automatically created group is set as 'legacy'.
    That way the scheduler knows that for this group it should emulate the
    old group behavior for backwards compat purposes.

    Partial-bug: #1296913

    Change-Id: I0d4315980af7e47ae97dd4277784a74f086c8825

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to nova (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/83724

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/83590
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2bca0c9011d9868aff3e80f4f6f432d29f07adf6
Submitter: Jenkins
Branch: master

commit 2bca0c9011d9868aff3e80f4f6f432d29f07adf6
Author: Russell Bryant <email address hidden>
Date: Thu Mar 27 20:40:23 2014 +0000

    Add new style instance group scheduler filters

    Prior to Icehouse, there was a different type of handling of the
    'group' scheduler hint that got lost in the completion of the server
    groups API. This patch completes the code necessary to provide
    backwards compatibility with the old behavior.

    Previously, the policy for groups was simply based on what scheduler
    filters you had enabled. You could have either the affinity or
    anti-affinity filter enabled and that would be applied to all groups.
    These filters now act on groups with a policy type of 'legacy'.

    New filters have been added that can be enabled simultaneously and act
    based on the policy set on the group via the server group API.

    DocImpact

    Change-Id: Ia65c151396415ca48725cb3c756f33efa01d2fe5
    Closes-bug: #1296913

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to nova (master)

Reviewed: https://review.openstack.org/83724
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7168940296236316de9f78953b2c39d0469626e6
Submitter: Jenkins
Branch: master

commit 7168940296236316de9f78953b2c39d0469626e6
Author: Russell Bryant <email address hidden>
Date: Fri Mar 28 11:52:19 2014 +0000

    Add a note about deprecated group filters

    The deprecated group filters do not work if both are enabled at the
    same time. Add this in a note in the docs on the different scheduler
    filters.

    DocImpact
    Related-bug: #1296913

    Change-Id: Ic0e1b8b59c8a462e193dffb3b0f873d8aa35aa3c

Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: icehouse-rc1 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.