Fails to boot instance using Blazar flavor if compute host names are in uppercase

[Neha Alhat]

Bug Description:
================

Steps to reproduce:
====================

$ nova hypervisor-list
+--------------------------------------+----------------------+--------------------+----------------------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+----------------------+--------------------+----------------------+
| e517e75b-d57c-45b2-af41-6be2fed536c6 | Openstack-VirtualBox | up | enabled |
+--------------------------------------+----------------------+--------------------+----------------------+

Step 1: Add host into freepool
$ blazar host-create Openstack-VirtualBox

$ blazar host-list
+----+---------------------+-------+-----------+----------+
| id | hypervisor_hostname | vcpus | memory_mb | local_gb |
+----+---------------------+-------+-----------+----------+
| 1 | Openstack-VirtualBox| 4 | 11941 | 91 |
+----+---------------------+-------+-----------+----------+

Step 2: Create a lease
$ blazar lease-create --reservation resource_type=virtual:instance,vcpus=1,memory_mb=1024,disk_gb=20,amount=1,affinity=False --start-date "2018-08-27 12:59" --end-date "2018-08-27 13:55" lease-1

nova_api database entries related to aggregates:

Blazar creates aggregate with id=18
mysql> select * from aggregates;
+---------------------+------------+----+--------------------------------------+--------------------------------------+
| created_at | updated_at | id | uuid | name |
+---------------------+------------+----+--------------------------------------+--------------------------------------+
| 2018-08-13 06:49:37 | NULL | 1 | 2a7d838f-4e42-48af-a9a9-faf3f29e3c96 | freepool |
| 2018-08-29 13:43:15 | NULL | 18 | 88c37cc5-373a-4da5-820f-508d25f00903 | 8c85522a-cc39-4a0d-a5ca-4de92e4d2c1f |
+---------------------+------------+----+--------------------------------------+--------------------------------------+

Blazar adds aggregate with aggregate_id=18, to host at the time of lease-start event.

mysql> select * from aggregate_hosts;
+---------------------+------------+----+-----------------------+--------------+
| created_at | updated_at | id | host | aggregate_id |
+---------------------+------------+----+-----------------------+--------------+
| 2018-08-29 13:34:46 | NULL | 32 | Openstack-VirtualBox | 1 |
| 2018-08-29 13:39:05 | NULL | 34 | Openstack-VirtualBox | 18 |
+---------------------+------------+----+-----------------------+--------------+

Step 3: Create a server: Please specify the flavor of the reservation and group_id as a scheduler hint.
$ openstack server create --flavor 03067174-2a5e-43f7-baf7-037aac23b4ef --image cirros-0.4.0-x86_64-disk --network 42d6f419-b445-40a6-b542-e8a502c6ae64 --hint group=09389292-6639-48d6-9709-045061f42ebf instance-1

For more details regarding instance reservation please refer: https://docs.openstack.org/blazar/latest/cli/instance-reservation.html

Logs
=====

Service logs of n-sch:
=======================
Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: INFO nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filter AggregateInstanceExtraSpecsFilter returned 0 hosts Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: DEBUG nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filtering removed all hosts for the request with instance ID '7e5aef57-fec6-4e25-8219-0e01aab4b9f3'. Filter results: [('RetryFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('AvailabilityZoneFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ComputeFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ComputeCapabilitiesFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ImagePropertiesFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ServerGroupAntiAffinityFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ServerGroupAffinityFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('SameHostFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('DifferentHostFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('AggregateInstanceExtraSpecsFilter', None)] {{(pid=22198) get_filtered_objects /opt/stack/nova/nova/filters.py:129}}
Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: INFO nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filtering removed all hosts for the request with instance ID '7e5aef57-fec6-4e25-8219-0e01aab4b9f3'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'ServerGroupAntiAffinityFilter: (start: 1, end: 1)', 'ServerGroupAffinityFilter: (start: 1, end: 1)', 'SameHostFilter: (start: 1, end: 1)', 'DifferentHostFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 0)']

Service logs of n-super-cond:
===============================
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: WARNING nova.scheduler.utils [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: Traceback (most recent call last):
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: return func(*args, **kwargs)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/opt/stack/nova/nova/scheduler/manager.py", line 169, in select_destinations
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: allocation_request_version, return_alternates)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 91, in select_destinations
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: allocation_request_version, return_alternates)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 256, in _schedule
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: claimed_instance_uuids)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 293, in _ensure_sufficient_hosts
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: raise exception.NoValidHost(reason=reason)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: NoValidHost: No valid host was found. There are not enough hosts available.
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: : NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: WARNING nova.scheduler.utils [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] [instance: 7e5aef57-fec6-4e25-8219-0e01aab4b9f3] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.

'AggregateInstanceExtraSpecsFilter' returns 0 hosts because at[1] host_state.aggregates is not returning aggregate with metadata created by Blazar. Lately, this patch [2] was merged in which HostManager._get_aggregates_info method, the host is converted into lower case and then it finds aggregates for that host in host_aggregates_map. But if the compute host name is in uppercase, it doesn’t add the host in host_aggregates_map in lowercase in _update_aggregates method hence _get_aggregates_info method doesn’t return the aggregate associated with the requested host.

Expected Result: Instance should be booted on the host "Openstack-VirtualBox"
Actual Result: NoValidHost_Remote: No valid host was found

[1] : https://github.com/openstack/nova/blob/master/nova/scheduler/filters/utils.py#L38
[2] : https://review.openstack.org/#/c/498334

[Matt Riedemann (mriedem)]

https://review.openstack.org/#/q/Iee4b9bbf412adfdc6fdc62ea3429fb960d6ac2a2 was just released in pike and queens as well.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/604898

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/605260

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/605266

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/605268

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/604898
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c9448cbdbf96e7436b13ac5c3a92addfc0f2f5a2
Submitter: Zuul
Branch: master

commit c9448cbdbf96e7436b13ac5c3a92addfc0f2f5a2
Author: Dan Smith <email address hidden>
Date: Mon Sep 24 13:28:29 2018 -0700

    Revert "Make host_aggregate_map dictionary case-insensitive"

    This reverts commit 0dc0db932e3ad5ad911f2072015cb9854f6e4e23.

    The original change caused our host state processing to be inconsistent
    with our own hypervisors API. Automation tooling that used our API
    to add hosts to aggregates would fail silently. We are reverting
    this and will propose a check on the aggregate host add action
    which will confirm the case-sensitive mapping of the host being
    added, which is what we should have done in the first place.

    Change-Id: Ibd44ba9de5680958f55f0ae6325cfc33dabadc4c
    Closes-Bug: #1793747

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605260
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c8ea8ed4662a045f4121df688b85471e410593ec
Submitter: Zuul
Branch: stable/rocky

commit c8ea8ed4662a045f4121df688b85471e410593ec
Author: Dan Smith <email address hidden>
Date: Mon Sep 24 13:28:29 2018 -0700

    Revert "Make host_aggregate_map dictionary case-insensitive"

    This reverts commit 0dc0db932e3ad5ad911f2072015cb9854f6e4e23.

    The original change caused our host state processing to be inconsistent
    with our own hypervisors API. Automation tooling that used our API
    to add hosts to aggregates would fail silently. We are reverting
    this and will propose a check on the aggregate host add action
    which will confirm the case-sensitive mapping of the host being
    added, which is what we should have done in the first place.

    Change-Id: Ibd44ba9de5680958f55f0ae6325cfc33dabadc4c
    Closes-Bug: #1793747
    (cherry picked from commit c9448cbdbf96e7436b13ac5c3a92addfc0f2f5a2)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605266
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=329780eb9282b05959e9c6159869ef1b4538030d
Submitter: Zuul
Branch: stable/queens

commit 329780eb9282b05959e9c6159869ef1b4538030d
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 25 19:10:48 2018 -0400

    Revert "Make host_aggregate_map dictionary case-insensitive"

    This reverts commit 2caf8f65e414732fb1ef1ac32a5884a48af6655d.

    The original change caused our host state processing to be inconsistent
    with our own hypervisors API. Automation tooling that used our API
    to add hosts to aggregates would fail silently. We are reverting
    this and will propose a check on the aggregate host add action
    which will confirm the case-sensitive mapping of the host being
    added, which is what we should have done in the first place.

    NOTE(mriedem): Since this is a revert of a backport to queens,
    we don't cherry pick the revert from rocky, but this is using
    the same change ID as the revert in rocky.

    Change-Id: Ibd44ba9de5680958f55f0ae6325cfc33dabadc4c
    Closes-Bug: #1793747

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.7 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605268
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c5cae2f63448c145c80b7f82ddccc7a9e672d534
Submitter: Zuul
Branch: stable/pike

commit c5cae2f63448c145c80b7f82ddccc7a9e672d534
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 25 19:22:22 2018 -0400

    Revert "Make host_aggregate_map dictionary case-insensitive"

    This reverts commit 8f5fbd83de3759407b8f928159067211af156d60.

    The original change caused our host state processing to be inconsistent
    with our own hypervisors API. Automation tooling that used our API
    to add hosts to aggregates would fail silently. We are reverting
    this and will propose a check on the aggregate host add action
    which will confirm the case-sensitive mapping of the host being
    added, which is what we should have done in the first place.

    NOTE(mriedem): Since this is a revert of a backport to pike,
    we don't cherry pick the revert from queens, but this is using
    the same change ID as the revert in queens.

    Change-Id: Ibd44ba9de5680958f55f0ae6325cfc33dabadc4c
    Closes-Bug: #1793747

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.