oslo.policy 3.6.1 breaks nova

[Stephen Finucane]

As seen on the requirements change [1], a recently introduced version of oslo.policy appears to be breaking nova [2]. Initial investigations suggest both oslo.policy and nova are partially to blame.

[1] https://review.opendev.org/c/openstack/requirements/+/773779
[2] https://d138d4f526b4feb9aa23-c0b1a48165a1318087e38ccc28dcb2b0.ssl.cf5.rackcdn.com/773779/1/check/cross-nova-functional/d9729b8/testr_results.html

[Stephen Finucane (stephenfinucane)]

Consider an arbitrary failing test from the job linked above, which can be run like so:

  $ tox -e functional-py38 -- \
      -n nova.tests.functional.regressions.test_bug_1522536.TestServerGet.test_id_overlap

Having added a large number of print debugs (patch attached - can be applied to oslo.config tag 3.6.1), we see the following from nova functional tests (at nova commit 726e2fd03b5dec5d6aa22a12a68a2f8d7c8952b3):

        File "/home/stephenfin/Development/openstack/nova/nova/tests/functional/regressions/test_bug_1522536.py", line 26, in setUp
  super(TestServerGet, self).setUp()
        File "/home/stephenfin/Development/openstack/nova/nova/test.py", line 264, in setUp
  self.policy = self.useFixture(policy_fixture.PolicyFixture())
        File "/home/stephenfin/Development/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/testtools/testcase.py", line 735, in useFixture
  fixture.setUp()
        File "/home/stephenfin/Development/openstack/nova/nova/tests/unit/policy_fixture.py", line 56, in setUp
  nova.policy.init(suppress_deprecation_warnings=True)
        File "/home/stephenfin/Development/openstack/nova/nova/policy.py", line 93, in init
  _ENFORCER.load_rules()

This is followed shortly thereafter by:

        File "/home/stephenfin/Development/openstack/nova/nova/tests/functional/regressions/test_bug_1522536.py", line 27, in setUp
  self.useFixture(policy_fixture.RealPolicyFixture())
        File "/home/stephenfin/Development/openstack/nova/.tox/functional-py38/lib/python3.8/site-packages/testtools/testcase.py", line 735, in useFixture
  fixture.setUp()
        File "/home/stephenfin/Development/openstack/nova/nova/tests/unit/policy_fixture.py", line 52, in setUp
  nova.policy.reset()
        File "/home/stephenfin/Development/openstack/nova/nova/policy.py", line 54, in reset
  _ENFORCER.clear()

From this we can see there are two fixtures being applied: the legacy 'PolicyFixture', which should really be deprecated and removed by now, and the modern 'RealPolicyFixture'. The sequence of these calls is important.

The call to 'PolicyFixture' attempts to 'reset()' nova's global policy engine instance [1] but isn't able to do so since it hasn't been initialized yet. It then 'init()'s this global policy engine [2]. Since the engine is now initialized, the 'RealPolicyFixture's call to 'reset()' will be permitted. The enforcer is deleted and then later initialized again in the subsequent 'init()' call. However, there is a difference in how we initialized the enforcer the second time around. In the first call from the legacy 'PolicyFixture', we initialized it with a sample file [3] that we created from some fake policy. However, in the second call from the 'RealPolicyFixture' we don't generate a fake policy file but rather point to 'etc/nova/policy.yaml' in tree, which doesn't actually exist [4][5] and therefore has nothing to load from.

This change in how we initialize the enforcer has a knock on effect. For reasons I haven't yet entirely groked - though I suspect it's to ensure the user gets exactly what they asked for without modifications - we only apply the deprecated check string via an OR *if* the rule wasn't loaded...

[Ghanshyam Mann (ghanshyammann)]

yeah, _deprecated_rule_handled is persistent on Rule object and first time it is set even deprecated rule is not added as OR check and next time _deprecated_rule_handled is True and skip the _handle_deprecated_rule completely and deprecated rules never got added as OR.

This way it will fix that problem.
https://review.opendev.org/c/openstack/oslo.policy/+/773950

In Nova, we should really remove all these fixtures themselves except the RealPolicyFixture's so that all test work on defaults.

[Ghanshyam Mann (ghanshyammann)]

just linking oslo bug for Rule modification things- https://bugs.launchpad.net/oslo.policy/+bug/1914095

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.policy 3.6.2 release.