Tests that need policy.json can never find it if run in isolation

[Julian Edwards]

I am writing some tests that inherit from RestfulTestCase. The server-side of the calls that the tests make are doing some checks that require policy.json to be read in.

However, it seems as though the file can't ever be found.

Looking at oslo_config, it searches these directories:

def _get_config_dirs(project=None):
    """Return a list of directories where config files may be located.

    :param project: an optional project name

    If a project is specified, following directories are returned::

      ~/.${project}/
      ~/
      /etc/${project}/
      /etc/

    Otherwise, these directories::

      ~/
      /etc/
    """

So basically under $HOME or /etc. The ConfigOpts.find_file() func also looks in any directory specified with the --config-dir option.

When running the tests, $HOME is set to a temp dir for each and every test, and --config-dir is not set at all. This currently seems to make it impossible for policy.json to be ever discovered in test runs.

[Julian Edwards (julian-edwards)]

Snippet from the server-side traceback:

  File "keystone/common/wsgi.py", line 329, in assert_admin
    self.policy_api.enforce(creds, 'admin_required', {})
  File "keystone/policy/backends/rules.py", line 77, in enforce
    enforce(credentials, action, target)
  File "keystone/policy/backends/rules.py", line 69, in enforce
    return _ENFORCER.enforce(action, target, credentials, **extra)
  File "/home/juledwar/mcopenstack/keystone/.tox/debug/local/lib/python2.7/site-packages/oslo_policy/policy.py", line 488, in enforce
    self.load_rules()
  File "/home/juledwar/mcopenstack/keystone/.tox/debug/local/lib/python2.7/site-packages/oslo_policy/policy.py", line 393, in load_rules
    self.policy_path = self._get_policy_path(self.policy_file)
  File "/home/juledwar/mcopenstack/keystone/.tox/debug/local/lib/python2.7/site-packages/oslo_policy/policy.py", line 461, in _get_policy_path
    raise cfg.ConfigFilesNotFoundError((path,))
ConfigFilesNotFoundError: Failed to find some config files: policy.json

[Julian Edwards (julian-edwards)]

Ok found the problem, I think.

oslo_policy.policy.Enforcer is initialised from the global config before the TestCase.setUp is called, which sets up policy_file in the config fixture.

So the Enforcer's policy_file just has 'policy.json' without the full path that is set up too late.

[ZhiQiang Fan (aji-zqfan)]

https://github.com/openstack/keystone/blob/8f30e9a0785234a5a9cf4daaa95207c9adb10835/keystone/tests/unit/core.py#L543

[Julian Edwards (julian-edwards)]

The Enforcer is initialised before config_overrides are called.

In my own setUp I can work around the problem with
keystone.policy.backends.rules._ENFORCER.policy_file = <blah>

[Julian Edwards (julian-edwards)]

I just noticed that in some test runs it is all OK. This might be a race condition or a test isolation issue - perhaps running some tests in isolation won't work because other tests are setting it up correctly?

[Lance Bragstad (lbragstad)]

Hi Julian,

The keystone unit tests set most of this stuff up in keystone/tests/unit/core.py, where it defines the location for various testing files [0] [1]. There are static methods available for accessing testing files as well [2]. The test_policy.py module in keystone has examples on how to access the policy file for testing in case you haven't used that as a reference yet [3].

Do you have a patch up for review we could look at? Let me know if this helps clear things up.

[0] https://github.com/openstack/keystone/blob/8dd27d3368ce0aa396386010f6aec5ed6304d687/keystone/tests/unit/core.py#L64-L70
[1] https://github.com/openstack/keystone/blob/8dd27d3368ce0aa396386010f6aec5ed6304d687/keystone/tests/unit/core.py#L95-L114
[2] https://github.com/openstack/keystone/blob/8dd27d3368ce0aa396386010f6aec5ed6304d687/keystone/tests/unit/core.py#L154
[3] https://github.com/openstack/keystone/blob/8dd27d3368ce0aa396386010f6aec5ed6304d687/keystone/tests/unit/test_policy.py#L199

[Julian Edwards (julian-edwards)]

Hey, thanks for the response Lance. I mostly understand how the policy file is loaded but I'm either missing something or there's a bug here somewhere.

It's some custom code that I sadly can't put up publicly, but I've narrowed things down to a simple case I think, and it's a problem in existing code. If you do this, you'll reproduce it on master:
tox -e py27 keystone.tests.unit.test_catalog

If I run *all* the tests, the tests will pass fine. If you run just these tests, the problem exists.

This shows that:
 - there is a test isolation problem
 - these tests are relying on the actions of previous tests which do something that cause the policy file to be loaded.

I'm not familiar enough with the code to understand what's causing the policy file to get loaded early enough when running all the tests though.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/278528

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/278528
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=10773b79ad7e9738ced9afe65071c60d92c8ca55
Submitter: Jenkins
Branch: master

commit 10773b79ad7e9738ced9afe65071c60d92c8ca55
Author: David Stanek <email address hidden>
Date: Wed Feb 10 16:06:03 2016 +0000

    Moves policy setup into a fixture.

    The original implemention worked only because of the ordering of the
    tests. By the time the tests that needed policy.json to be loaded ran it
    had already been properly loaded. When running certain tests in
    isolation the policy is not propertly setup, leading to a test failure.

    Closes-Bug: #1520383
    Change-Id: Icd041eb4ed8ddd580f49b4709ca5f05ab7315292

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/keystone 9.0.0.0b3 development milestone.