Fails to boot instance using Blazar flavor if compute host names are in uppercase

Bug #1793747 reported by Neha Alhat on 2018-09-21
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
High
Dan Smith
Pike
High
Matt Riedemann
Queens
High
Matt Riedemann
Rocky
High
Matt Riedemann

Bug Description

Bug Description:
================

Steps to reproduce:
====================

$ nova hypervisor-list
+--------------------------------------+----------------------+--------------------+----------------------+
| ID | Hypervisor hostname | State | Status |
+--------------------------------------+----------------------+--------------------+----------------------+
| e517e75b-d57c-45b2-af41-6be2fed536c6 | Openstack-VirtualBox | up | enabled |
+--------------------------------------+----------------------+--------------------+----------------------+

Step 1: Add host into freepool
$ blazar host-create Openstack-VirtualBox

$ blazar host-list
+----+---------------------+-------+-----------+----------+
| id | hypervisor_hostname | vcpus | memory_mb | local_gb |
+----+---------------------+-------+-----------+----------+
| 1 | Openstack-VirtualBox| 4 | 11941 | 91 |
+----+---------------------+-------+-----------+----------+

Step 2: Create a lease
$ blazar lease-create --reservation resource_type=virtual:instance,vcpus=1,memory_mb=1024,disk_gb=20,amount=1,affinity=False --start-date "2018-08-27 12:59" --end-date "2018-08-27 13:55" lease-1

nova_api database entries related to aggregates:

Blazar creates aggregate with id=18
mysql> select * from aggregates;
+---------------------+------------+----+--------------------------------------+--------------------------------------+
| created_at | updated_at | id | uuid | name |
+---------------------+------------+----+--------------------------------------+--------------------------------------+
| 2018-08-13 06:49:37 | NULL | 1 | 2a7d838f-4e42-48af-a9a9-faf3f29e3c96 | freepool |
| 2018-08-29 13:43:15 | NULL | 18 | 88c37cc5-373a-4da5-820f-508d25f00903 | 8c85522a-cc39-4a0d-a5ca-4de92e4d2c1f |
+---------------------+------------+----+--------------------------------------+--------------------------------------+

Blazar adds aggregate with aggregate_id=18, to host at the time of lease-start event.

mysql> select * from aggregate_hosts;
+---------------------+------------+----+-----------------------+--------------+
| created_at | updated_at | id | host | aggregate_id |
+---------------------+------------+----+-----------------------+--------------+
| 2018-08-29 13:34:46 | NULL | 32 | Openstack-VirtualBox | 1 |
| 2018-08-29 13:39:05 | NULL | 34 | Openstack-VirtualBox | 18 |
+---------------------+------------+----+-----------------------+--------------+

Step 3: Create a server: Please specify the flavor of the reservation and group_id as a scheduler hint.
$ openstack server create --flavor 03067174-2a5e-43f7-baf7-037aac23b4ef --image cirros-0.4.0-x86_64-disk --network 42d6f419-b445-40a6-b542-e8a502c6ae64 --hint group=09389292-6639-48d6-9709-045061f42ebf instance-1

For more details regarding instance reservation please refer: https://docs.openstack.org/blazar/latest/cli/instance-reservation.html

Logs
=====

Service logs of n-sch:
=======================
Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: INFO nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filter AggregateInstanceExtraSpecsFilter returned 0 hosts Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: DEBUG nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filtering removed all hosts for the request with instance ID '7e5aef57-fec6-4e25-8219-0e01aab4b9f3'. Filter results: [('RetryFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('AvailabilityZoneFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ComputeFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ComputeCapabilitiesFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ImagePropertiesFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ServerGroupAntiAffinityFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('ServerGroupAffinityFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('SameHostFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('DifferentHostFilter', [(u'Openstack-VirtualBox', u'Openstack-VirtualBox')]), ('AggregateInstanceExtraSpecsFilter', None)] {{(pid=22198) get_filtered_objects /opt/stack/nova/nova/filters.py:129}}
Aug 27 13:00:31 Openstack-VirtualBox nova-scheduler[22177]: INFO nova.filters [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Filtering removed all hosts for the request with instance ID '7e5aef57-fec6-4e25-8219-0e01aab4b9f3'. Filter results: ['RetryFilter: (start: 1, end: 1)', 'AvailabilityZoneFilter: (start: 1, end: 1)', 'ComputeFilter: (start: 1, end: 1)', 'ComputeCapabilitiesFilter: (start: 1, end: 1)', 'ImagePropertiesFilter: (start: 1, end: 1)', 'ServerGroupAntiAffinityFilter: (start: 1, end: 1)', 'ServerGroupAffinityFilter: (start: 1, end: 1)', 'SameHostFilter: (start: 1, end: 1)', 'DifferentHostFilter: (start: 1, end: 1)', 'AggregateInstanceExtraSpecsFilter: (start: 1, end: 0)']

Service logs of n-super-cond:
===============================
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: WARNING nova.scheduler.utils [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: Traceback (most recent call last):
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 226, in inner
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: return func(*args, **kwargs)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/opt/stack/nova/nova/scheduler/manager.py", line 169, in select_destinations
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: allocation_request_version, return_alternates)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 91, in select_destinations
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: allocation_request_version, return_alternates)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 256, in _schedule
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: claimed_instance_uuids)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 293, in _ensure_sufficient_hosts
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: raise exception.NoValidHost(reason=reason)
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: NoValidHost: No valid host was found. There are not enough hosts available.
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: : NoValidHost_Remote: No valid host was found. There are not enough hosts available.
Aug 27 13:00:31 Openstack-VirtualBox nova-conductor[22131]: WARNING nova.scheduler.utils [None req-c1891de9-d8b9-4d3f-ae78-9b8ad848a5ec admin admin] [instance: 7e5aef57-fec6-4e25-8219-0e01aab4b9f3] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.

'AggregateInstanceExtraSpecsFilter' returns 0 hosts because at[1] host_state.aggregates is not returning aggregate with metadata created by Blazar. Lately, this patch [2] was merged in which HostManager._get_aggregates_info method, the host is converted into lower case and then it finds aggregates for that host in host_aggregates_map. But if the compute host name is in uppercase, it doesn’t add the host in host_aggregates_map in lowercase in _update_aggregates method hence _get_aggregates_info method doesn’t return the aggregate associated with the requested host.

Expected Result: Instance should be booted on the host "Openstack-VirtualBox"
Actual Result: NoValidHost_Remote: No valid host was found

[1] : https://github.com/openstack/nova/blob/master/nova/scheduler/filters/utils.py#L38
[2] : https://review.openstack.org/#/c/498334

Neha Alhat (nehaalhat) on 2018-09-21
Changed in nova:
assignee: nobody → Neha Alhat (nehaalhat)
Matt Riedemann (mriedem) on 2018-09-22
tags: added: scheduler
tags: added: aggregates
Changed in nova:
status: New → Triaged
importance: Undecided → High
Matt Riedemann (mriedem) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/604898

Changed in nova:
assignee: Neha Alhat (nehaalhat) → Dan Smith (danms)
status: Triaged → In Progress
Changed in nova:
assignee: Dan Smith (danms) → Matt Riedemann (mriedem)
Matt Riedemann (mriedem) on 2018-09-25
Changed in nova:
assignee: Matt Riedemann (mriedem) → Dan Smith (danms)

Reviewed: https://review.openstack.org/604898
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c9448cbdbf96e7436b13ac5c3a92addfc0f2f5a2
Submitter: Zuul
Branch: master

commit c9448cbdbf96e7436b13ac5c3a92addfc0f2f5a2
Author: Dan Smith <email address hidden>
Date: Mon Sep 24 13:28:29 2018 -0700

    Revert "Make host_aggregate_map dictionary case-insensitive"

    This reverts commit 0dc0db932e3ad5ad911f2072015cb9854f6e4e23.

    The original change caused our host state processing to be inconsistent
    with our own hypervisors API. Automation tooling that used our API
    to add hosts to aggregates would fail silently. We are reverting
    this and will propose a check on the aggregate host add action
    which will confirm the case-sensitive mapping of the host being
    added, which is what we should have done in the first place.

    Change-Id: Ibd44ba9de5680958f55f0ae6325cfc33dabadc4c
    Closes-Bug: #1793747

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/605260
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c8ea8ed4662a045f4121df688b85471e410593ec
Submitter: Zuul
Branch: stable/rocky

commit c8ea8ed4662a045f4121df688b85471e410593ec
Author: Dan Smith <email address hidden>
Date: Mon Sep 24 13:28:29 2018 -0700

    Revert "Make host_aggregate_map dictionary case-insensitive"

    This reverts commit 0dc0db932e3ad5ad911f2072015cb9854f6e4e23.

    The original change caused our host state processing to be inconsistent
    with our own hypervisors API. Automation tooling that used our API
    to add hosts to aggregates would fail silently. We are reverting
    this and will propose a check on the aggregate host add action
    which will confirm the case-sensitive mapping of the host being
    added, which is what we should have done in the first place.

    Change-Id: Ibd44ba9de5680958f55f0ae6325cfc33dabadc4c
    Closes-Bug: #1793747
    (cherry picked from commit c9448cbdbf96e7436b13ac5c3a92addfc0f2f5a2)

Reviewed: https://review.openstack.org/605266
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=329780eb9282b05959e9c6159869ef1b4538030d
Submitter: Zuul
Branch: stable/queens

commit 329780eb9282b05959e9c6159869ef1b4538030d
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 25 19:10:48 2018 -0400

    Revert "Make host_aggregate_map dictionary case-insensitive"

    This reverts commit 2caf8f65e414732fb1ef1ac32a5884a48af6655d.

    The original change caused our host state processing to be inconsistent
    with our own hypervisors API. Automation tooling that used our API
    to add hosts to aggregates would fail silently. We are reverting
    this and will propose a check on the aggregate host add action
    which will confirm the case-sensitive mapping of the host being
    added, which is what we should have done in the first place.

    NOTE(mriedem): Since this is a revert of a backport to queens,
    we don't cherry pick the revert from rocky, but this is using
    the same change ID as the revert in rocky.

    Change-Id: Ibd44ba9de5680958f55f0ae6325cfc33dabadc4c
    Closes-Bug: #1793747

This issue was fixed in the openstack/nova 18.0.2 release.

This issue was fixed in the openstack/nova 17.0.7 release.

Reviewed: https://review.openstack.org/605268
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c5cae2f63448c145c80b7f82ddccc7a9e672d534
Submitter: Zuul
Branch: stable/pike

commit c5cae2f63448c145c80b7f82ddccc7a9e672d534
Author: Matt Riedemann <email address hidden>
Date: Tue Sep 25 19:22:22 2018 -0400

    Revert "Make host_aggregate_map dictionary case-insensitive"

    This reverts commit 8f5fbd83de3759407b8f928159067211af156d60.

    The original change caused our host state processing to be inconsistent
    with our own hypervisors API. Automation tooling that used our API
    to add hosts to aggregates would fail silently. We are reverting
    this and will propose a check on the aggregate host add action
    which will confirm the case-sensitive mapping of the host being
    added, which is what we should have done in the first place.

    NOTE(mriedem): Since this is a revert of a backport to pike,
    we don't cherry pick the revert from queens, but this is using
    the same change ID as the revert in queens.

    Change-Id: Ibd44ba9de5680958f55f0ae6325cfc33dabadc4c
    Closes-Bug: #1793747

This issue was fixed in the openstack/nova 16.1.7 release.

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers