Addition of host to host-aggregate should be case -sensitive

[Rajesh Tailor]

I have multinode devstack setup with 2 compute and a controller node.
The hostnames of the nodes are as follows:
controller: controller.example.com
compute0: compute0.example.com
compute1: compute1.example.com

Reproduction steps:

1) I have created a host-aggregate in availability zone "nova".
    nova aggregate-create test_agg nova

2) Then I set metadata on that host-aggregate.
    nova aggregate-set-metadata 1 ssd=true

3) Then I added compute hosts mentioned above to host-aggregate.
    nova aggregate-add-host test_agg COMPUTE0.example.com
    nova aggregate-add-host test_agg compute1.example.com

While adding compute0 to host-aggregate, if I provide hostname as "COMPUTE0.example.com", instead of
throwing HostNotFound error, it is added to host-aggregate.

4) After that I have created a flavor.
nova flavor-create ssd.large 6 8192 80 4

5) Added metadata to flavor.
nova flavor-key ssd.large set ssd=true

Now when I try to create instance with above mentioned flavor, I can't spawn it on compute0 host. As scheduler is filtering only compute1 with AggregateInstanceExtraSpecsFilter filter.

Expected Result: while adding the case-sensitive host to host-aggregate, nova-api should have thrown HostNotFound error.

Note: when I add the correct hostname "compute0.example.com", I am able to spawn instance on with above mentioned flavor on compute0 host and it is filtered by AggregateInstanceExtraSpecFilter as well.

[Sylvain Bauza (sylvain-bauza)]

For clarity on the bug report, could you please provide the nova-sceduler log that shows the filter missing that host because the name is different ?

TBC, all FQDNs are case-sensitive, so we should probably indeed return a 404 when you're trying to add a host with the wrong case. The only point I wonder is whether it would require a new API microversion since we were previously accepting that from a REST perpective (my guts tend to say yes to that :( )

[Rajesh Tailor (ratailor)]

Hi Sylvain,

I have debugged the issue and found that IMO it has nothing to do with scheduler.

While adding host to host-aggregate, the below db query should not matched to existing db record.
Since mysql doesn't support case-sensitive column.

The problematic block of code is below:
https://github.com/openstack/nova/blob/master/nova/objects/host_mapping.py#L90-L93

    def _get_by_host_from_db(context, host):
        db_mapping = (context.session.query(api_models.HostMapping)
                      .options(joinedload('cell_mapping'))
                      .filter(api_models.HostMapping.host == host)).first()

[Alex Xu (xuhj)]

I think it is the problem of inconsistent behavior between mysql DB and the in-memory cache in the nova scheduler.

The mysql Db doesn't support case-sensitive filtering, so the API layer can ensure the host existed. But there is in-memory cache build in the scheduler https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L359, it becomes case-senstive for the key of a dict.

[Rajesh Tailor (ratailor)]

Please find the attached n-sch logs for more information.

[Alex Xu (xuhj)]

related bug https://bugs.launchpad.net/nova/+bug/1538011

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/498334

[Jay Pipes (jaypipes)]

For the record, MySQL *does* support case-sensitive matching. It's just not the default.

Adding case-sensitive matching is as simple as changing the default collation to latin1_swedish_cs instead of latin1_swedish_ci. Or utf8_general_cs instead of utf8_general_ci if the character set on the column in question is UTF8.

[Jay Pipes (jaypipes)]

According to https://github.com/openstack/nova/blob/c0350da4a1607d7aa113caceaefb5d29303c7eed/nova/db/sqlalchemy/api_models.py#L53, there is no character set specified for the aggregate_hosts.host field (and there is no default in the oslo_db.sqlalchemy.models.ModelBase class) so the default character set will be latin1_swedish_ci.

It would be a fairly massive undertaking to "fix" this in all of Nova. We'd have to add default character set + collation clauses to all models and add a single DB schema migration that would change the character set and collation for all columns in question. This would require an offline data migration since it wouldn't be a happy instantaneous schema change. So, we'd have to add duplicate columns with the "correct" collation, migrate/copy all the old data into the new columns, then delete the old columns.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/585334

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/585335

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/498334
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0dc0db932e3ad5ad911f2072015cb9854f6e4e23
Submitter: Zuul
Branch: master

commit 0dc0db932e3ad5ad911f2072015cb9854f6e4e23
Author: Rajesh Tailor <email address hidden>
Date: Fri Aug 18 12:33:09 2017 +0530

    Make host_aggregate_map dictionary case-insensitive

    As of now, if hostname is set as "compute0.example.com" (in lower case)
    and user tries to add this host to host-aggregate but by-mistake types
    "COMPUTE0.example.com" (in capital case), then instead of throwing
    HostNotFound error, that host is successfully added to host-aggregate
    as "COMPUTE0.example.com". And after that if instance creation request
    with flavor matching metadata as host-aggregate metadata comes, this
    host is not filtered by scheduler, since there is no host with hostname
    COMPUTE0.example.com, as added in host-aggregate.

    Fixed the issue by lowercasing all hostnames in host_aggregate_map
    dictionary.

    Change-Id: Iee4b9bbf412adfdc6fdc62ea3429fb960d6ac2a2
    Closes-Bug: 1709260

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585334
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=2caf8f65e414732fb1ef1ac32a5884a48af6655d
Submitter: Zuul
Branch: stable/queens

commit 2caf8f65e414732fb1ef1ac32a5884a48af6655d
Author: Rajesh Tailor <email address hidden>
Date: Fri Aug 18 12:33:09 2017 +0530

    Make host_aggregate_map dictionary case-insensitive

    As of now, if hostname is set as "compute0.example.com" (in lower case)
    and user tries to add this host to host-aggregate but by-mistake types
    "COMPUTE0.example.com" (in capital case), then instead of throwing
    HostNotFound error, that host is successfully added to host-aggregate
    as "COMPUTE0.example.com". And after that if instance creation request
    with flavor matching metadata as host-aggregate metadata comes, this
    host is not filtered by scheduler, since there is no host with hostname
    COMPUTE0.example.com, as added in host-aggregate.

    Fixed the issue by lowercasing all hostnames in host_aggregate_map
    dictionary.

    Change-Id: Iee4b9bbf412adfdc6fdc62ea3429fb960d6ac2a2
    Closes-Bug: 1709260
    (cherry picked from commit 0dc0db932e3ad5ad911f2072015cb9854f6e4e23)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/585335
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8f5fbd83de3759407b8f928159067211af156d60
Submitter: Zuul
Branch: stable/pike

commit 8f5fbd83de3759407b8f928159067211af156d60
Author: Rajesh Tailor <email address hidden>
Date: Fri Aug 18 12:33:09 2017 +0530

    Make host_aggregate_map dictionary case-insensitive

    As of now, if hostname is set as "compute0.example.com" (in lower case)
    and user tries to add this host to host-aggregate but by-mistake types
    "COMPUTE0.example.com" (in capital case), then instead of throwing
    HostNotFound error, that host is successfully added to host-aggregate
    as "COMPUTE0.example.com". And after that if instance creation request
    with flavor matching metadata as host-aggregate metadata comes, this
    host is not filtered by scheduler, since there is no host with hostname
    COMPUTE0.example.com, as added in host-aggregate.

    Fixed the issue by lowercasing all hostnames in host_aggregate_map
    dictionary.

    Change-Id: Iee4b9bbf412adfdc6fdc62ea3429fb960d6ac2a2
    Closes-Bug: 1709260
    (cherry picked from commit 0dc0db932e3ad5ad911f2072015cb9854f6e4e23)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.6 release.

[Matt Riedemann (mriedem)]

Note the change is being reverted https://review.openstack.org/#/c/604898/ due to it causing a regression, see bug 1793747.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/604906

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/605261

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/605267

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/605269

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/604906
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c8e2de668434861b87495e0a0f715b2f90d8ec05
Submitter: Zuul
Branch: master

commit c8e2de668434861b87495e0a0f715b2f90d8ec05
Author: Dan Smith <email address hidden>
Date: Mon Sep 24 13:36:39 2018 -0700

    Enforce case-sensitive hostnames in aggregate host add

    If we are using a case-insensitive backend database like mysql, a
    user can request an aggregate host add with a non-matching hostname
    and we will not signal failure. We will create a mapping which will
    not actually include that host in the aggregate, which will be
    confusing later. This change makes us fail if the host name does not
    match exactly, which is the same behavior as if we are using a
    case-sensitive backend (like postgres).

    Change-Id: I597dee74d33de337913eddda74ab056fbf81a23c
    Closes-Bug: #1709260

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605261
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c72bb2c747016b22e71109b9750d5899b776e67a
Submitter: Zuul
Branch: stable/rocky

commit c72bb2c747016b22e71109b9750d5899b776e67a
Author: Dan Smith <email address hidden>
Date: Mon Sep 24 13:36:39 2018 -0700

    Enforce case-sensitive hostnames in aggregate host add

    If we are using a case-insensitive backend database like mysql, a
    user can request an aggregate host add with a non-matching hostname
    and we will not signal failure. We will create a mapping which will
    not actually include that host in the aggregate, which will be
    confusing later. This change makes us fail if the host name does not
    match exactly, which is the same behavior as if we are using a
    case-sensitive backend (like postgres).

    Change-Id: I597dee74d33de337913eddda74ab056fbf81a23c
    Closes-Bug: #1709260
    (cherry picked from commit c8e2de668434861b87495e0a0f715b2f90d8ec05)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605267
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=22197f3d0156b9fb87a132e77ab44dbdd63bc826
Submitter: Zuul
Branch: stable/queens

commit 22197f3d0156b9fb87a132e77ab44dbdd63bc826
Author: Dan Smith <email address hidden>
Date: Mon Sep 24 13:36:39 2018 -0700

    Enforce case-sensitive hostnames in aggregate host add

    If we are using a case-insensitive backend database like mysql, a
    user can request an aggregate host add with a non-matching hostname
    and we will not signal failure. We will create a mapping which will
    not actually include that host in the aggregate, which will be
    confusing later. This change makes us fail if the host name does not
    match exactly, which is the same behavior as if we are using a
    case-sensitive backend (like postgres).

    Conflicts:
          nova/tests/unit/compute/test_compute.py
          nova/tests/unit/compute/test_host_api.py

    NOTE(mriedem): Conflicts are due to not having change
    Ibd7aa4f8c4ea787774becece324d9051521c44b6 in Queens.

    Change-Id: I597dee74d33de337913eddda74ab056fbf81a23c
    Closes-Bug: #1709260
    (cherry picked from commit c8e2de668434861b87495e0a0f715b2f90d8ec05)
    (cherry picked from commit c72bb2c747016b22e71109b9750d5899b776e67a)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.7 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/605269
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=0bd38dbadcaeef59213e5471018649f7a9d2d9bf
Submitter: Zuul
Branch: stable/pike

commit 0bd38dbadcaeef59213e5471018649f7a9d2d9bf
Author: Dan Smith <email address hidden>
Date: Mon Sep 24 13:36:39 2018 -0700

    Enforce case-sensitive hostnames in aggregate host add

    If we are using a case-insensitive backend database like mysql, a
    user can request an aggregate host add with a non-matching hostname
    and we will not signal failure. We will create a mapping which will
    not actually include that host in the aggregate, which will be
    confusing later. This change makes us fail if the host name does not
    match exactly, which is the same behavior as if we are using a
    case-sensitive backend (like postgres).

    Change-Id: I597dee74d33de337913eddda74ab056fbf81a23c
    Closes-Bug: #1709260
    (cherry picked from commit c8e2de668434861b87495e0a0f715b2f90d8ec05)
    (cherry picked from commit c72bb2c747016b22e71109b9750d5899b776e67a)
    (cherry picked from commit 22197f3d0156b9fb87a132e77ab44dbdd63bc826)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.