A bad _RC_CACHE can rarely cause unit tests to fail

Bug #1747003 reported by Chris Dent
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Chris Dent

Bug Description

Very rarely (so rarely in fact that it only seems to happen when test order is much different from the norm) some unit tests which encounter the resource_class_cache can fail as follows:

http://logs.openstack.org/49/540049/2/check/openstack-tox-py27/176a6b3/testr_results.html.gz

-=-=-
ft1.1: nova.tests.unit.cmd.test_status.TestUpgradeCheckResourceProviders.test_check_resource_providers_no_compute_rps_one_compute_StringException: pythonlogging:'': {{{2018-02-02 11:30:00,443 WARNING [oslo_config.cfg] Config option key_manager.api_class is deprecated. Use option key_manager.backend instead.}}}

Traceback (most recent call last):
  File "nova/tests/unit/cmd/test_status.py", line 588, in test_check_resource_providers_no_compute_rps_one_compute
    self._create_resource_provider(FAKE_IP_POOL_INVENTORY)
  File "nova/tests/unit/cmd/test_status.py", line 561, in _create_resource_provider
    rp.set_inventory(inv_list)
  File "nova/api/openstack/placement/objects/resource_provider.py", line 737, in set_inventory
    exceeded = _set_inventory(self._context, self, inv_list)
  File "/home/zuul/src/git.openstack.org/openstack/nova/.tox/py27/local/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 986, in wrapper
    return fn(*args, **kwargs)
  File "nova/api/openstack/placement/objects/resource_provider.py", line 372, in _set_inventory
    _add_inventory_to_provider(context, rp, inv_list, to_add)
  File "nova/api/openstack/placement/objects/resource_provider.py", line 201, in _add_inventory_to_provider
    if inv_record.capacity <= 0:
AttributeError: 'NoneType' object has no attribute 'capacity'
-=-=-

The find() method on InventoryList can return None if that cache is bad.

This can be resolved (apparently) by resetting the _RC_CACHE between test runs in the same way that _TRAITS_SYNCED is reset, in nova/test.py:

- # Reset the traits sync flag
- objects.resource_provider._TRAITS_SYNCED = False
+ # Reset the traits sync flag and rc cache
+ resource_provider._TRAITS_SYNCED = False
+ resource_provider._RC_CACHE = None

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/540404

Changed in nova:
assignee: nobody → Chris Dent (cdent)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/540404
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=95ad6a2a9af8833baf69f025cad185e8bc857653
Submitter: Zuul
Branch: master

commit 95ad6a2a9af8833baf69f025cad185e8bc857653
Author: Chris Dent <email address hidden>
Date: Fri Feb 2 14:40:44 2018 +0000

    Reset the _RC_CACHE between tests

    Very rarely the _RC_CACHE used for caching ResourceClass id and name
    mappings will be wrong for tests, resulting in the find() method on
    InventoryList returning None leading to inv.capacity calls failing.

    This only showed up in the tests for
    Iea182341f9419cb514a044f76864d6bec60a3683 where the order of tests
    are changed because of a lot of renaming of modules.

    Change-Id: Idc318d3914fa600deff613d8a43eadd9073fa262
    Closes-Bug: #1747003

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 17.0.0.0rc1

This issue was fixed in the openstack/nova 17.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.