Cannot get ComputeNodeStat by DB utility of compute_node_get_all()

[Yong Feng]

When there is hypervisor gets removed, the compute_node_get_all() will not return stat for new added hypervisors.

In the following codes of compute_node_get_all() of nova/db/sqlalchemy, it assume all the record in compute_node_stats should have a matched compute node. However in current implementation of nova conductor API of compute_node_delete(), the records in compute_node_stats is not deleted. Therefore when a hypervisor gets removed, there is no node matching the record of compute_node_stats which belongs to the removed hypervisor in following codes. As a result, all the nodes will be set with 'stats' of [].

    # Join ComputeNode & ComputeNodeStat manually.
    # NOTE(msdubov): ComputeNode and ComputeNodeStat map 1-to-Many.
    # Running time is (asymptotically) optimal due to the use
    # of iterators (itertools.groupby() for ComputeNodeStat and
    # iter() for ComputeNode) - we handle each record only once.
    compute_nodes.sort(key=lambda node: node['id'])
    compute_nodes_iter = iter(compute_nodes)
    for nid, nsts in itertools.groupby(stats, lambda s: s['compute_node_id']):
        for node in compute_nodes_iter:
            if node['id'] == nid:
                node['stats'] = list(nsts)
                break
            else:
                node['stats'] = []

    return compute_nodes

We need enhance either nova conductor API to clean up all the record related with instance.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/46379

[Joe Cropper (jwcroppe)]

Will try to take a look at this.

[Joe Cropper (jwcroppe)]

After further investigation, this does seem to behave appropriately. I've gone ahead and added some test cases to show that deleting either the compute node or service (two new test cases) does indeed cascade the deletes to the compute node stats.

[Joe Cropper (jwcroppe)]

Scratch my previous comment, I misunderstood what the compute_node_statistics call returned. After closer inspection, this does seem to be problematic. Continuing investigation.

[Joe Cropper (jwcroppe)]

Fix and test case has been submitted.

Thanks,
Joe

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/46379
Committed: http://github.com/openstack/nova/commit/8b9d6f6fedbbd47932cd672f51d4db7031724e84
Submitter: Jenkins
Branch: master

commit 8b9d6f6fedbbd47932cd672f51d4db7031724e84
Author: Joe Cropper <email address hidden>
Date: Thu Sep 12 17:16:47 2013 -0500

    Prune node stats at compute node delete time

    This commit addresses the situation in which a compute node is deleted, but
    its compute node stats still remain in the database as **not** deleted. This
    causes ill side effects in compute_node_get_all when it's retrieving host stats
    as it doesn't expect there to be compute node stats for which there is no
    corresponding compute node (i.e., causing some nodes' stats to be empty).

    As such, when a compute node is deleted, its stats should also be implicitly
    deleted.

    The new test case that's been created fails without the code changes, which
    illustrates the problem that compute node stats are empty when they should
    not be.

    Also included is a simple DB migration script that will update old stats that
    were not marked soft-deleted as they should have been.

    Change-Id: Ief0f7cf1a506e71898b5a45a0513d34167432d67
    Closes-Bug: #1224712