Compute schedulers in nova - AggregateInstanceExtraSpecsFilter docs are not clear

[Matt Riedemann]

- [x] This is a doc addition request.

The description for the AggregateInstanceExtraSpecsFilter filter is not clear:

https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregateinstanceextraspecsfilter

(note it's also described here: https://docs.openstack.org/nova/latest/user/filter-scheduler.html)

It's not clear what aggregate_instance_extra_specs is used for.

Note that further down in the document there are some examples:

https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#example-specify-compute-
hosts-with-ssds

So I guess based on that, it means you would just add metadata to a host aggregate like foo=bar and then tie a flavor to that by setting an extra spec of aggregate_instance_extra_specs:foo=bar on the flavor. But what about other standard extra specs like hide_hypervisor_id, you can't put the aggregate_instance_extra_specs prefix on that in the flavor since it would break the extra spec for the actual code that checks for that standard extra spec. Does that mean the flavor has to have both the scoped and unscoped spec? Or that the filter will handle the unscoped spec? It would be nice to have the documentation on the filter itself explain this and give examples of how to use it, for both a standard and custom flavor extra spec (note the latter has an example linked above for the ssd example).

This originally came up while triaging bug 1841932 and trying to make sense of the filter (it's not very clear even by looking at the code).

-----------------------------------
Release: on 2019-08-22 20:13:47
SHA: 0882ea69ea0c46cf97ecd5a1ec49a3007f293c28
Source: https://opendev.org/openstack/nova/src/doc/source/admin/configuration/schedulers.rst
URL: https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html

[sean mooney (sean-k-mooney)]

no the flavor does not have to have scoped and unsocped forms of hide_hypervisor_id

the flavor just needt to have hide_hypervisor_id

and an aggreate needs to be create with hide_hypervisor_id=true or false in the metadata
to match the flavor value.

the AggregateInstanceExtraSpecsFilter checks that all extra_specs that are unscoped or
are scoped with aggregate_instance_extra_specs in the flavor are present in at least 1 aggregate that
the host is a member of.

if the host is a member of 2 aggreates
one of which has hide_hypervisor_id=true and the second of which has hide_hypervisor_id=false

then flavor with both hide_hypervisor_id=true or hide_hypervisor_id=false will be valid for this host. the filter does not support logical operators so if you want to or values together then you must use multipel aggreates.

this out of tree filter which is not relly maintained is geenerally more useful
https://github.com/openstack/nfv-filters/blob/master/nfv_filters/nova/scheduler/filters/aggregate_instance_type_filter.py
https://github.com/openstack/nfv-filters/blob/master/doc/source/scheduler_filters/aggregate-instance-type-filter.rst

but the real solution goign forward is likely to use CUSTOM_TRAITS and placmeent instead.

the usecasue of this filter was to do partioning of you cloud but the intree filter is really easy to misconfigure.

for example if you add hide_hypervisor_id=true to an aggrate but the flavor does not request it
then the filter will still pass the host which is not what people expect.

the the canonical example of where this goes wrong is people creating an NFV aggregate with and seeing NFV=true in there aggreate and in ther NFV flavours and then being confused by why there standard flavours that dont have NFV set in the flavor land in there NFV aggreate.

the out of tree filter invert the relationship by instead requireing that all extra_spec in the aggreate metadta must be in the flavor instead. which is whay you normally want as it will only allw the NFV flavor to go to the NFV aggrate without having to add NFV=false to all your standard flavors which is required whne using the in tree AggregateInstanceExtraSpecsFilter

[Matt Riedemann (mriedem)]

Thanks Sean. Part of your comment is also describing https://specs.openstack.org/openstack/nova-specs/specs/train/approved/placement-req-filter-forbidden-aggregates.html.

[Matt Riedemann (mriedem)]

Bug 1634032 is also an example of confusion over how this filter works.

[kiran pawar (kpdev)]

Ideally AggregateInstanceExtraSpecsFilter should check only scoped specs in aggregate metadata.
e.g.

flavor spec {"key1": "value1}" and {"aggregate_instance_extra_specs:key1":"value1"}, both are filtered to must have {"key1": "value1"} in either of host aggregate. This should be limited to only scoped spec.

Is it bug in filter or its expected behavior ?

[sean mooney (sean-k-mooney)]

it is expected behavior not a bug.

support for matching on unscoped keys was intentional.
removing it would be an upgrade impact so we would have to deprecate it for a release before we could remove it. currently we discourage using unscoped extra specs in general regardless fo the filter but its still allowed so until we remove that capableity in nova its reasonable for the AggregateInstanceExtraSpecsFilter to be able to match on them.

long term i would prefer if non standard extra specs were required to start with custom:
or another namespace but that is not the case today.

hide_hypervisor_id specifically has been deprecated and should no longer be used.

we introduced hw:hide_hypervisor_id as a replacement to explicitly avoid the conflict with the AggregateInstanceExtraSpecsFilter
https://github.com/openstack/nova/commit/bf488a8630702160021b5848bf6e86fbb8015205

[sean mooney (sean-k-mooney)]

we can discuss deprecating the unscoped support or the filter in general as we now have better alternatives such as https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html

but given the upgrade impact this would need a deprecation cycle at a minium to notify operators that they will need to resize all there instances to update the flavor which is not cheap or we will need to provide tooling such as nova-manage commands or similar to help with that.

that falls out of the scope of a simple bug fix in my view as we would have to come to a consensus on how to change the behavior which likely should be done via a spec or (specless blueprint and mailing list/ptg discussion)

given the above setting this to wont fix for now.