The quota_usage_refresh counts wrong when there exists soft-deleted instances

[huanhongda]

Description
===========
When there exists soft-deleted instances in my openstack, executing "nova-manage project quota_usage_refresh" will get wrong values in quota_usages.

Steps to reproduce
==================
1) Set reclaim_instance_interval>0 to enable soft-delete;
2) Boot a new instance, then check quota_usages tables. Assume instances' in_use is 1, ram's in_use is 512, cores' in_use is 4.
3) Delete that instance, and it's status will migrate to SOFT_DELETED.
4) Check quota_usages tables. Both instances, ram and cores in_use is 0.
5) Execute "nova-manage project quota_usage_refresh --project <project-id> --user <user-id>".
6) Check quota_usages tables. The instances' in_use is 1, ram's in_use is 512, cores' in_use is 4.

Expected result
===============
After the execution of quota_usage_refresh, both instances, ram and cores in_use should be 0.

Actual result
=============
The instances' in_use is 1, ram's in_use is 512, cores' in_use is 4. That's because quota_usage_refresh will count all instances whose "deleted" field is 0 in "instances" table. And soft-deleted instance's "deleted" field is 0, so it will be counted as in used.

Environment
===========
This bug was found in Ocata.
Pike, Queens and master don't have this bug.
The quota_usage_refresh command has been removed in 17.0.0 (Queens). Since quota usage is counted from resources instead of being tracked separately.
https://docs.openstack.org/nova/pike/cli/nova-manage.html

And they have filtered out soft-deleted instances while counting from resources:
https://github.com/openstack/nova/blob/5bf9aa7fd455e93b73dc13aba66ccb7ec00f23a5/nova/objects/instance.py#L1475-L1489

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/570653

[jichenjc (jichenjc)]

assign to hongda as already a fix there

[OpenStack Infra (hudson-openstack)]

Change abandoned by huanhongda (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/570653
Reason: Submit to master first then cherry pick.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/570736

[OpenStack Infra (hudson-openstack)]

Change abandoned by huanhongda (<email address hidden>) on branch: master
Review: https://review.openstack.org/570736
Reason: Submit this patch to stable/ocata directly.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/579093

[OpenStack Infra (hudson-openstack)]

Change abandoned by huanhongda (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/570653
Reason: Fix the bug in stable/queens first then backport to stable/ocata.

[melanie witt (melwitt)]

This is invalid for Rocky, so marking the Rocky series version of the bug as Invalid.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/602459

[OpenStack Infra (hudson-openstack)]

Change abandoned by Lee Yarwood (<email address hidden>) on branch: stable/pike
Review: https://review.openstack.org/602459
Reason: Missed the stable/ocata change was already closed.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/579093
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=58538ace357d97bd9a06ce042aadc2928b1ff5bc
Submitter: Zuul
Branch: stable/queens

commit 58538ace357d97bd9a06ce042aadc2928b1ff5bc
Author: huanhongda <email address hidden>
Date: Sat May 26 22:40:27 2018 +0800

    [Stable Only] Remove soft-deleted instances from quota_usages

    The resources in quota_usages table will minus 1 while
    soft deleting an instance. But the soft-deleted instances
    are counted when synchronizing quota_usages using command
    "nova-manage project quota_usage_refresh". So we filter
    out soft-deleted instances when synchronizing quota_usage.

    We didn't need to do this on master as the quota_usage_refresh
    command has been removed in 17.0.0 (Queens). And the
    _instance_data_get_for_user function is already removed
    in commit c881e77819f1b8ca48c0dfd0ebfb75daaabf1033.
    https://review.openstack.org/#/c/570865/

    Change-Id: I8dcbe1e5f3f70ddd7067fed1125ef3379044ab0f
    Closes-bug: #1773542

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/602459
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ab03100bcbd5afe5702b760057fc19e25300cd89
Submitter: Zuul
Branch: stable/pike

commit ab03100bcbd5afe5702b760057fc19e25300cd89
Author: huanhongda <email address hidden>
Date: Sat May 26 22:40:27 2018 +0800

    [Stable Only] Remove soft-deleted instances from quota_usages

    The resources in quota_usages table will minus 1 while
    soft deleting an instance. But the soft-deleted instances
    are counted when synchronizing quota_usages using command
    "nova-manage project quota_usage_refresh". So we filter
    out soft-deleted instances when synchronizing quota_usage.

    We didn't need to do this on master as the quota_usage_refresh
    command has been removed in 17.0.0 (Queens). And the
    _instance_data_get_for_user function is already removed
    in commit c881e77819f1b8ca48c0dfd0ebfb75daaabf1033.
    https://review.openstack.org/#/c/570865/

    Change-Id: I8dcbe1e5f3f70ddd7067fed1125ef3379044ab0f
    Closes-bug: #1773542
    (cherry picked from commit 58538ace357d97bd9a06ce042aadc2928b1ff5bc)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.6 release.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/570653
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b4659017936d415c4395a57c52fad6cf3ffa5302
Submitter: Zuul
Branch: stable/ocata

commit b4659017936d415c4395a57c52fad6cf3ffa5302
Author: huanhongda <email address hidden>
Date: Sat May 26 22:40:27 2018 +0800

    [Stable Only] Remove soft-deleted instances from quota_usages

    The resources in quota_usages table will minus 1 while
    soft deleting an instance. But the soft-deleted instances
    are counted when synchronizing quota_usages using command
    "nova-manage project quota_usage_refresh". So we filter
    out soft-deleted instances when synchronizing quota_usage.

    We didn't need to do this on master as the quota_usage_refresh
    command has been removed in 17.0.0 (Queens). And the
    _instance_data_get_for_user function is already removed
    in commit c881e77819f1b8ca48c0dfd0ebfb75daaabf1033.
    https://review.openstack.org/#/c/570865/

    Change-Id: I8dcbe1e5f3f70ddd7067fed1125ef3379044ab0f
    Closes-bug: #1773542
    (cherry picked from commit 58538ace357d97bd9a06ce042aadc2928b1ff5bc)
    (cherry picked from commit ab03100bcbd5afe5702b760057fc19e25300cd89)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.5 release.