Placement request filter behaves differently with AggregateMultiTenancyIsolation

[Shi Yan]

Description
===========

When enabling flag limit_tenants_to_placement_aggregate, we found the scheduling results will be different with existing AggregateMultiTenancyIsolation scheduler filter.

Steps to reproduce
==================

Consider the below scenario, we have defined two aggregates:
Aggr1 (without define any metadata)
Aggr2 (define metadata filter_tenant_id to include our working projects)

So for the old AggregateMultiTenancyIsolation scheduler filter, hosts in *both Aggr* will be considered for the instance launching if the incoming requests are from our working projects.
But if we enable limit_tenants_to_placement_aggregate, only Aggr2 will be sent to placement for the allocation candidates query. Then the following scheduling will only consider the hosts in *Aggr2*.

Such behavior is aligned with the code does now (nova/scheduler/request_filter.py : require_tenant_aggregate) but the document does not state such difference clearly, or maybe such placement req filter behavior needs to change?

== extracted from the doc about the placement req filter ===
This setting causes the scheduler to look up a host aggregate with the metadata key of filter_tenant_id set to the project of an incoming request, and request results from placement be limited to that aggregate. Multiple tenants may be added to a single aggregate by appending a serial number to the key, such as filter_tenant_id:123.

[Balazs Gibizer (balazs-gibizer)]

This doc feels relevant even if it talks about the scheduler.enable_isolated_aggregate_filtering: https://docs.openstack.org/nova/latest/reference/isolate-aggregates.html

I think the observed behavior is correct and the documentation could be improved.