Listing placement usages (total or per resource provider) in a new process can result in a 500

[Chris Dent]

When requesting /usages or /resource_providers/{uuid}/usages it is possible to cause a 500 error if placement is running in a multi-process scenario and the usages query is the first request a process has received. This is because the methods which provide UsageLists do not _ensure_rc_cache, resulting in:

  File "/usr/lib/python3.6/site-packages/nova/api/openstack/placement/objects/resource_provider.py", line 2374, in _from_db_object
       rc_str = _RC_CACHE.string_from_id(source['resource_class_id'])
   AttributeError: 'NoneType' object has no attribute 'string_from_id'

We presumably don't see this in our usual testing because any process has already had other requests happen, setting the cache.

For now, the fix is to add the _ensure_rc_cache call in the right places, but long term if/when we switch to the os-resource-class model we can do the caching or syncing a bit differently (see https://review.openstack.org/#/c/553857/ for an example).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/572652

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/572652
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=724d440122232a5bfdfec51eb0d37ca4f1d748d8
Submitter: Zuul
Branch: master

commit 724d440122232a5bfdfec51eb0d37ca4f1d748d8
Author: Chris Dent <email address hidden>
Date: Tue Jun 5 18:07:05 2018 -0700

    Ensure resource class cache when listing usages

    In rare circumstances it is possible to list usages in a new placement
    process that has not yet instantiated the _RC_CACHE but for which
    there are inventories and allocations in the database (added by
    other processes running against the same db). Before this change
    that would cause a 500 error (AttributeError) when the Usage objects
    in the UsageList were instantiated.

    The fix is to added _ensure_rc_cache to the two list methods. The
    addition is done there rather than in the _from_db_object as the
    latter would cause a lot of redundant checks.

    While we could probably devise a test for this, it's perhaps good
    enough to evaluate the change by inspection. If not, suggestions
    welcome.

    Change-Id: I00f7dee26f031366dbc0d3d6a03abe89afeb85fd
    Closes-Bug: #1775308

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 18.0.0.0b2 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/573811

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/573811
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=79a1a113775df87d420b242cea4bbf953df3779c
Submitter: Zuul
Branch: stable/queens

commit 79a1a113775df87d420b242cea4bbf953df3779c
Author: Chris Dent <email address hidden>
Date: Tue Jun 5 18:07:05 2018 -0700

    Ensure resource class cache when listing usages

    In rare circumstances it is possible to list usages in a new placement
    process that has not yet instantiated the _RC_CACHE but for which
    there are inventories and allocations in the database (added by
    other processes running against the same db). Before this change
    that would cause a 500 error (AttributeError) when the Usage objects
    in the UsageList were instantiated.

    The fix is to added _ensure_rc_cache to the two list methods. The
    addition is done there rather than in the _from_db_object as the
    latter would cause a lot of redundant checks.

    While we could probably devise a test for this, it's perhaps good
    enough to evaluate the change by inspection. If not, suggestions
    welcome.

    Change-Id: I00f7dee26f031366dbc0d3d6a03abe89afeb85fd
    Closes-Bug: #1775308
    (cherry picked from commit 724d440122232a5bfdfec51eb0d37ca4f1d748d8)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/574279

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chris Dent (<email address hidden>) on branch: stable/ocata
Review: https://review.openstack.org/574279
Reason: will learn the alphabet and then redo

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/574327

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/574327
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6e1dd287f18e03428b6f991c5b4bdd66901f8c2d
Submitter: Zuul
Branch: stable/pike

commit 6e1dd287f18e03428b6f991c5b4bdd66901f8c2d
Author: Chris Dent <email address hidden>
Date: Tue Jun 5 18:07:05 2018 -0700

    Ensure resource class cache when listing usages

    In rare circumstances it is possible to list usages in a new placement
    process that has not yet instantiated the _RC_CACHE but for which
    there are inventories and allocations in the database (added by
    other processes running against the same db). Before this change
    that would cause a 500 error (AttributeError) when the Usage objects
    in the UsageList were instantiated.

    The fix is to added _ensure_rc_cache to the two list methods. The
    addition is done there rather than in the _from_db_object as the
    latter would cause a lot of redundant checks.

    While we could probably devise a test for this, it's perhaps good
    enough to evaluate the change by inspection. If not, suggestions
    welcome.

    Change-Id: I00f7dee26f031366dbc0d3d6a03abe89afeb85fd
    Closes-Bug: #1775308
    (cherry picked from commit 724d440122232a5bfdfec51eb0d37ca4f1d748d8)
    (cherry picked from commit 79a1a113775df87d420b242cea4bbf953df3779c)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/574279
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=4bd2a8d9d97ba8ce2e1bab404454aeb543561760
Submitter: Zuul
Branch: stable/ocata

commit 4bd2a8d9d97ba8ce2e1bab404454aeb543561760
Author: Chris Dent <email address hidden>
Date: Tue Jun 5 18:07:05 2018 -0700

    Ensure resource class cache when listing usages

    In rare circumstances it is possible to list usages in a new placement
    process that has not yet instantiated the _RC_CACHE but for which
    there are inventories and allocations in the database (added by
    other processes running against the same db). Before this change
    that would cause a 500 error (AttributeError) when the Usage objects
    in the UsageList were instantiated.

    The fix is to added _ensure_rc_cache to the two list methods. The
    addition is done there rather than in the _from_db_object as the
    latter would cause a lot of redundant checks.

    While we could probably devise a test for this, it's perhaps good
    enough to evaluate the change by inspection. If not, suggestions
    welcome.

    This, the ocata version, has only half the changes provided by
    commits in newer versions because in ocata, get_all_by_project_user
    is not yet available.

    Change-Id: I00f7dee26f031366dbc0d3d6a03abe89afeb85fd
    Closes-Bug: #1775308
    (cherry picked from commit 724d440122232a5bfdfec51eb0d37ca4f1d748d8)
    (cherry picked from commit 79a1a113775df87d420b242cea4bbf953df3779c)
    (cherry picked from commit 6e1dd287f18e03428b6f991c5b4bdd66901f8c2d)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 16.1.5 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.6 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 15.1.4 release.