Filter AggregateImagePropertiesIsolation doesn't Work

[ByungYeol Woo]

Description
===========
I tried to use filter AggregateImagePropertiesIsolation to isolate Windows instance for reducing number of Windows Licenses.

I think nova scheduler in pike release, filter AggregateImagePropertiesIsolation always returned all hosts. If this is a bug, filter AggregateImagePropertiesIsolation needs to be fixed.

Steps to reproduce
==================
# add filter to nova.conf and restart nova scheduler
[filter_scheduler]
enabled_filters = AggregateImagePropertiesIsolation,RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter

# image create with os property
openstack image create --min-disk 3 --min-ram 512 --disk-format qcow2 --public --file windows.img img_windows
openstack image create --min-disk 1 --min-ram 64 --disk-format qcow2 --public --file cirros-0.3.5-x86_64-disk.img img_linux
openstack image set --property os=windows img_windows
openstack image set --property os=linux img_linux

# host aggregate create with os property
openstack aggregate create os_win
openstack aggregate add host os_win compute01
openstack aggregate add host os_win compute02
openstack aggregate set --property os=windows os_win

openstack aggregate create os_linux
openstack aggregate add host os_linux compute03
openstack aggregate add host os_linux compute04
openstack aggregate add host os_linux compute05
openstack aggregate set --property os=linux os_linux

# create flavor
openstack flavor create --ram 1024 --disk 1 --vcpus 1 --public small
openstack flavor create --ram 4096 --disk 20 --vcpus 2 --public medium

# create windows instances
openstack server create --image img_windows --network test-net --flavor medium --max 10 test-win

Expected result
===============
Windows instances can be found in compute01, compute02 only

Actual result
=============
Windows instance was found in every hosts.

Environment
===========
1. Nova's version
(nova-scheduler)[nova@control01 /]$ rpm -qa | grep nova
python-nova-17.0.0-0.20171206190932.cbdc893.el7.centos.noarch
openstack-nova-scheduler-17.0.0-0.20171206190932.cbdc893.el7.centos.noarch
openstack-nova-common-17.0.0-0.20171206190932.cbdc893.el7.centos.noarch
python2-novaclient-9.1.0-0.20170804194758.0a53d19.el7.centos.noarch

2. hypervisor
(nova-libvirt)[root@compute01 /]# rpm -qa | grep kvm
qemu-kvm-common-ev-2.9.0-16.el7_4.11.1.x86_64
libvirt-daemon-kvm-3.2.0-14.el7_4.5.x86_64
qemu-kvm-ev-2.9.0-16.el7_4.11.1.x86_64

2. Storage
ceph version 12.2.1 (3e7492b9ada8bdc9a5cd0feafd42fbca27f9c38e) luminous (stable)

3. Networking
Neutron with OpenVSwitch

Logs & Configs
==============
$ tail -f nova-scheduler.log | grep AggregateImagePropertiesIsolation
2018-01-08 11:52:53.964 6 DEBUG nova.filters [req-3828686f-1d46-407a-bebb-14f7a573c52e 9b1f4f0bcea2428c93b8b4276ba67cb7 188be4011b2b49529cbdd6eade152233 - default default] Filter AggregateImagePropertiesIsolation returned 5 host(s) get_filtered_objects /usr/lib/python2.7/site-packages/nova/filters.py:104

# add filter to nova.conf and restart nova scheduler
[filter_scheduler]
enabled_filters = AggregateImagePropertiesIsolation,RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter

[Takashi Natsume (natsume-takashi)]

You should set the following parameters in nova configuration file (nova.conf).
(They are in 'filter_scheduler' section.)

* aggregate_image_properties_isolation_namespace
* aggregate_image_properties_isolation_separator

[ByungYeol Woo (wby1089)]

I tried to use those parameters like following:

[filter_scheduler]
enabled_filters = AggregateImagePropertiesIsolation,RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter
aggregate_image_properties_isolation_namespace = testimg
aggregate_image_properties_isolation_separator = .

[openstack cli]
openstack image set --property testimg:os=windows img_windows
openstack image set --property testimg:os=linux img_linux

openstack aggregate set --property os=windows os_win
openstack aggregate set --property os=windows os_win

But result was not diffrent. AggregateImagePropertiesIsolation also didn't work.

[Takashi Natsume (natsume-takashi)]

> You should set the following parameters in nova configuration file (nova.conf).

Sorry. It is not correct.

> aggregate_image_properties_isolation_namespace = testimg
> aggregate_image_properties_isolation_separator = .

But in this case, it may work well by the following operations.

openstack image set --property testimg.os=windows img_windows
openstack image set --property testimg.os=linux img_linux

openstack aggregate set --property testimg.os=windows os_win
openstack aggregate set --property testimg.os=linux os_linux

If it does not work, needs more information.
For example:

* The output of 'openstack image show' command
* The output of 'openstack aggregate show' command
* nova.conf file that nova-scheduler uses
* the log file of nova-scheduler
* nova/scheduler/filters/aggregate_image_properties_isolation.py file
* The value of 'x-openstack-request-id' when creating a server (POST /compute/v2.1/servers)
  (You can get it by 'debug' option. e.g. "openstack --debug server create")

[ByungYeol Woo (wby1089)]

Thanks for kind informations.

I changed property of image and aggregate as 'testimg.os=windows', but it doesn't work also.
And I found 'nximg.os' was not present in the image metadata by nova scheduller's log.

============================================
2018-01-12 09:36:53.359 7 WARNING nova.scheduler.filters.aggregate_image_properties_isolation [req-7b802c30-490f-4874-ba34-51353d596de2 cbe8b47b44de4549a203193fb8b99a57 38e39fba2d3f4f08a6ebfdd60e73df54 - default default] Host 'compute02' has a metadata key 'nximg.os' that is not present in the image metadata.
2018-01-12 09:36:53.360 7 WARNING nova.scheduler.filters.aggregate_image_properties_isolation [req-7b802c30-490f-4874-ba34-51353d596de2 cbe8b47b44de4549a203193fb8b99a57 38e39fba2d3f4f08a6ebfdd60e73df54 - default default] Host 'compute01' has a metadata key 'nximg.os' that is not present in the image metadata.
============================================

Please see attached file.

[Felix Ma (felix23ma)]

I have the same issue on Pike release.

Analysis:
========
Nova gets image property like 'testimg.os=linux' from glance, but it's dropped when nova creates req_spec here:
https://github.com/openstack/nova/blob/stable/pike/nova/compute/api.py#L899

It seems from_dict in ImageMetaProps class only keeps properties which have key name predefined in source code: (_legacy_property_map and self.fields)
https://github.com/openstack/nova/blob/stable/pike/nova/objects/image_meta.py#L552
========

My configuration is same as ByungYeol Woo. It works fine on Kilo. I'm not sure if any other configuration is missed.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[Kosal Raj I (kosalraj)]

Is the issue still exists for the pike release ?

[ByungYeol Woo (wby1089)]

The issue is still exists in ocata, pike and queens release.

[Kosal Raj I (kosalraj)]

Thank you for the update.

[ByungYeol Woo (wby1089)]

I found the condition of this issue.

If I set property as "os_type=windows" or "os_distro=windows", it worked well.
But If I set property as "os=windows" or "os_asdf=windows", it didsn't work.

<worked property>
os_type=windows
os_distro=windows

<no worked property>
os=windows
my_test_prop=windows
os_asdf=windows

I think AggregateImagePropertiesIsolation filter works well when property is well-known filter.

[Sean Matheny (smat924)]

Any update on this? I can confirm this bug / behaviour still exists in Stein. E.g.:

(venv) sean@ntr-ops01:~> openstack aggregate show image-filtered
+-------------------+----------------------------+
| Field | Value |
+-------------------+----------------------------+
| availability_zone | nova |
| created_at | 2020-07-01T01:09:27.000000 |
| deleted | False |
| deleted_at | None |
| hosts | uc01 |
| id | 3 |
| name | image-filtered |
| properties | os_distro='arch' |
| updated_at | None |
+-------------------+----------------------------+
(venv) sean@ntr-ops01:~> openstack aggregate show image-unfiltered
+-------------------+----------------------------+
| Field | Value |
+-------------------+----------------------------+
| availability_zone | nova |
| created_at | 2020-07-01T01:28:55.000000 |
| deleted | False |
| deleted_at | None |
| hosts | uc02, uc03 |
| id | 4 |
| name | image-unfiltered |
| properties | os_distro='windows' |
| updated_at | None |
+-------------------+----------------------------+

and for image:
| properties | os_distro='arch',

Works, but anything not predefined in https://docs.openstack.org/glance/stein/admin/useful-image-properties.html doesn't seem to work.

When I use a non-predefined key, I get these warnings in nova-scheduler for all hosts (even though the key is present):

Host 'uc01' has a metadata key 'imagetype' that is not present in the image metadata.

In this example:
-image has a metadata pair: imagetype='gpu'
-aggregates have metadata pair: imagetype='gpu' or imagetype='generic'

Launching an instance matches all 3 hosts in both aggregates by AggregateImagePropertiesIsolation filter:

2020-07-01 13:54:43.621 12493 DEBUG nova.filters [req-adc01bed-3a71-4ba1-89d2-a306befc9fc9 5ade285fd6644de4a92fc4133482b90b ef21d7642978447981ee9b7721f7a448 - default default] Filter AggregateImagePropertiesIsolation returned 3 host(s) get_filtered_objects /usr/lib/python3/dist-packages/nova/filters.py:104

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/752236

[Stephen Finucane (stephenfinucane)]

We discussed this on IRC today [1]. In short, we realize that this was a change in behaviour introduced in Liberty that should have been better discussed at the time. However, Liberty was many years ago and it's genuinely debatable whether this was ever intended behaviour, let alone something we'd want to reintroduce support for.

Having discussed this, we're going to document this change in behaviour in the docs and leave it there. If this (support for arbitrary image metadata properties in this filter) is something you still see value in, we'd probably have to treat it as a new feature. I'd encourage you to file a spec [2] so we can evaluate the idea. If not, hopefully the documentation change helps clarify things.

[1] http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2020-09-16.log.html#t2020-09-16T13:18:37
[2] https://specs.openstack.org/openstack/nova-specs/readme.html

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/752236
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2a7f1f986e3575f41a791fef93c35197b0075dab
Submitter: Zuul
Branch: master

commit 2a7f1f986e3575f41a791fef93c35197b0075dab
Author: Stephen Finucane <email address hidden>
Date: Wed Sep 16 13:47:13 2020 +0100

    docs: Improve 'AggregateImagePropertiesIsolation' docs

    When we introduced the 'ImageMetaProps' o.vo in Liberty, we lost the
    ability to consume arbitrary metadata configured for images. This
    affects users of the 'AggregateImagePropertiesIsolation' filter, who may
    have set such arbitrary metadata and expected their instances to be
    restricted to host aggregates matching that metadata.

    The world has changed a lot since Liberty was released, and it's
    probably too late and maybe even a little unwise to let that genie back
    out of its bottle, however, we can and should probably do a better job
    of warning people of this change in behavior in our documentation. Do
    just this.

    Change-Id: If7245a90711bd2ea13095ba26b9bc82ea3e17202
    Signed-off-by: Stephen Finucane <email address hidden>
    Related-Bug: #1741810