compute resource providers not equal to compute nodes in deployment

[Hanif Kukkalli]

Description
===========
When I execute the command `nova-status upgrade check`,
output:

nova-status upgrade check
+----------------------------------------------------------------------+
| Upgrade Check Results |
+----------------------------------------------------------------------+
| Check: Cells v2 |
| Result: Success |
| Details: None |
+----------------------------------------------------------------------+
| Check: Placement API |
| Result: Success |
| Details: None |
+----------------------------------------------------------------------+
| Check: Resource Providers |
| Result: Warning |
| Details: There are 4 compute resource providers and 15 compute nodes |
| in the deployment. Ideally the number of compute resource |
| providers should equal the number of enabled compute nodes |
| otherwise the cloud may be underutilized. See |
| http://docs.openstack.org/developer/nova/placement.html |
| for more details. |
+----------------------------------------------------------------------+

Steps to reproduce
==================
How to replicate this?
Remove the hosts from the openstack controller:
nova hypervisor-list
nova service-delete {id}

Then run:
su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova
The deleted compute node will be added again as a new node.
run:
nova-status upgrade check

Expected result
===============
No warning when you run:
nova-status upgrade check

Actual result
=============
You can find the warning.
This causes issue with placement of new VM's.
The compute host which was deleted and added again will not be considered during VM scheduling and placement.

Environment
===========
OpenStack Pike release
Neutron Networking which is default.

Logs and Configs
================
Config as the Openstack documentation.

[Matt Riedemann (mriedem)]

> su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova
The deleted compute node will be added again as a new node.

That doesn't actually have anything to do with the compute node records getting re-created after you delete the service records.

Are you actually stopping the nova-compute service before you delete the service records via the API? Because if not, there is a periodic task in the nova-compute service that will automatically re-create the compute_nodes records in the database.

See the note in the API reference:

https://developer.openstack.org/api-ref/compute/#delete-compute-service

[Matt Riedemann (mriedem)]

This is also related:

https://bugs.launchpad.net/nova/+bug/1756179

[Hanif Kukkalli (kukkalli)]

Hi Matt,

I had stopped the nova-compute service.
The issue is with the Placement API which created duplicates of the records.

[Launchpad Janitor (janitor)]

[Expired for OpenStack Compute (nova) because there has been no activity for 60 days.]

[iain MacDonnell (imacdonn)]

Shouldn't the upgrade check simply discount deleted compute_nodes ?

+-----------------------------------------------------------------------+
| Check: Resource Providers |
| Result: Warning |
| Details: There are 34 compute resource providers and 40 compute nodes |
| in the deployment. Ideally the number of compute resource |
| providers should equal the number of enabled compute nodes |
| otherwise the cloud may be underutilized. See |
| https://docs.openstack.org/nova/latest/user/placement.html |
| for more details. |
+-----------------------------------------------------------------------+

mysql> select count(*) from compute_nodes;
+----------+
| count(*) |
+----------+
| 40 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from compute_nodes where deleted=0;
+----------+
| count(*) |
+----------+
| 34 |
+----------+
1 row in set (0.00 sec)

[Chris Dent (cdent)]

It appears that not accounting for deleted is discussed here: https://github.com/openstack/nova/blob/6be7f7248fb1c2bbb890a0a48a424e205e173c9c/nova/cmd/status.py#L114-L128

It does seem a bit weird.

[iain MacDonnell (imacdonn)]

"disabled" is one thing - you can come back from that - but "deleted" is a one-way ticket ... no?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/604495

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/604495
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3e902313e9c3117e76d6adf88a2f104bc229374c
Submitter: Zuul
Branch: master

commit 3e902313e9c3117e76d6adf88a2f104bc229374c
Author: imacdonn <email address hidden>
Date: Sat Sep 22 00:55:36 2018 +0000

    nova-status - don't count deleted compute_nodes

    When counting rows in the compute_nodes table to compare against
    resource records, those which have been marked as deleted should
    not be counted, otherwise the result is artificially high.

    Change-Id: Ia3fbb9c46c15826fc363286ce48984f59cbe1e62
    Closes-Bug: #1757207

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/604785

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/604786

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/604788

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/604785
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c559c5b357f47ae2bed9397bbfb8b2e08dc5b8a5
Submitter: Zuul
Branch: stable/rocky

commit c559c5b357f47ae2bed9397bbfb8b2e08dc5b8a5
Author: imacdonn <email address hidden>
Date: Sat Sep 22 00:55:36 2018 +0000

    nova-status - don't count deleted compute_nodes

    When counting rows in the compute_nodes table to compare against
    resource records, those which have been marked as deleted should
    not be counted, otherwise the result is artificially high.

    Change-Id: Ia3fbb9c46c15826fc363286ce48984f59cbe1e62
    Closes-Bug: #1757207
    (cherry picked from commit 3e902313e9c3117e76d6adf88a2f104bc229374c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/604786
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=72d2e82e1275035c70cb6862ff15dc41b3d9b621
Submitter: Zuul
Branch: stable/queens

commit 72d2e82e1275035c70cb6862ff15dc41b3d9b621
Author: imacdonn <email address hidden>
Date: Sat Sep 22 00:55:36 2018 +0000

    nova-status - don't count deleted compute_nodes

    When counting rows in the compute_nodes table to compare against
    resource records, those which have been marked as deleted should
    not be counted, otherwise the result is artificially high.

    Change-Id: Ia3fbb9c46c15826fc363286ce48984f59cbe1e62
    Closes-Bug: #1757207
    (cherry picked from commit 3e902313e9c3117e76d6adf88a2f104bc229374c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/604788
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ef5c02209a28510eee66edc8495acc6cd4e0993a
Submitter: Zuul
Branch: stable/pike

commit ef5c02209a28510eee66edc8495acc6cd4e0993a
Author: imacdonn <email address hidden>
Date: Sat Sep 22 00:55:36 2018 +0000

    nova-status - don't count deleted compute_nodes

    When counting rows in the compute_nodes table to compare against
    resource records, those which have been marked as deleted should
    not be counted, otherwise the result is artificially high.

    Change-Id: Ia3fbb9c46c15826fc363286ce48984f59cbe1e62
    Closes-Bug: #1757207
    (cherry picked from commit 3e902313e9c3117e76d6adf88a2f104bc229374c)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.2 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 19.0.0.0rc1 release candidate.