List resource providers consistently fail with 500 error

[Takashi Kajinami]

I initially reported this in https://storyboard.openstack.org/#!/story/2010108 but it seems the issue is specific to Ubuntu packages so I'm reporting this bug for the packaging side as well.

This issue was found in puppet integration job in ubuntu focal + yoga.
placement 1:7.0.0-0ubuntu1~cloud0
python3 3.8.2-0ubuntu2
python3-oslo.policy 3.11.0-0ubuntu1~cloud0

List resource providers API consistently returns 500 error, and nova-compute can't start because of that API error.
Looking at placement api log, the following traceback is found.

```
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap [req-b2bf439f-9e22-4d16-8cd2-a191b0070e19 acf1133c873a4427b4ca8b3d026af72f 77a64ffe39e340b480fdb3c45adb5273 - default default] Placement API unexpected error: unsupported callable: TypeError: unsupported callable
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap Traceback (most recent call last):
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3.8/inspect.py", line 1135, in getfullargspec
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap sig = _signature_from_callable(func,
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3.8/inspect.py", line 2233, in _signature_from_callable
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap sig = _signature_from_callable(
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3.8/inspect.py", line 2304, in _signature_from_callable
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap return _signature_from_function(sigcls, obj,
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3.8/inspect.py", line 2168, in _signature_from_function
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap parameters.append(Parameter(name, annotation=annotation,
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3.8/inspect.py", line 2491, in __init__
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap self._kind = _ParameterKind(kind)
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3.8/enum.py", line 339, in __call__
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap return cls.__new__(cls, value)
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3.8/enum.py", line 635, in __new__
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap if type(value) is cls:
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap RecursionError: maximum recursion depth exceeded while calling a Python object
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap The above exception was the direct cause of the following exception:
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap Traceback (most recent call last):
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/placement/fault_wrap.py", line 39, in __call__
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap return self.application(environ, start_response)
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap resp = self.call_func(req, *args, **kw)
...
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/placement/handlers/allocation.py", line 299, in list_for_resource_provider
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap context.can(policies.RP_ALLOC_LIST)
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/placement/context.py", line 56, in can
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap return policy.authorize(self, action, target)
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/placement/policy.py", line 115, in authorize
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap LOG.debug('Policy check for %(action)s failed with credentials '
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap self.force_reraise()
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap raise self.value
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/placement/policy.py", line 104, in authorize
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap return _ENFORCER.authorize(
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 1173, in authorize
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap return self.enforce(
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 1071, in enforce
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap result = _checks._check(
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 80, in _check
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap return rule(*rule_args)
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 257, in __call__
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap return _check(
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 80, in _check
2022-06-23 23:32:46.277 26465 ERROR placement.fault_wrap return rule(*rule_args)
...
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 80, in _check
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap return rule(*rule_args)
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 213, in __call__
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap if _check(rule, target, cred, enforcer, current_rule):
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 80, in _check
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap return rule(*rule_args)
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 257, in __call__
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap return _check(
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 80, in _check
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap return rule(*rule_args)
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 257, in __call__
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap return _check(
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap File "/usr/lib/python3/dist-packages/oslo_policy/_checks.py", line 75, in _check
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap argspec = inspect.getfullargspec(rule.__call__)
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap File "/usr/lib/python3.8/inspect.py", line 1144, in getfullargspec
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap raise TypeError('unsupported callable') from ex
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap TypeError: unsupported callable
2022-06-23 23:32:46.311 26466 ERROR placement.fault_wrap
```

The same issue is observed in stable/xena job as well.

[Takashi Kajinami (kajinamit)]

I don't know how files in ubuntu packages are generated by the strange thing is that the /etc/policy/policy.yaml file contains the following three lines WITHOUT heading #, which looks completely invalid.

https://zuul.opendev.org/t/openstack/build/9e1d5c6e77c848ce88a753030ffa6d02

https://145b0d5be77f0cbe05ff-d67a6226f6eb6d34ab4efd951744a16f.ssl.cf5.rackcdn.com/854538/1/check/puppet-openstack-integration-6-scenario002-tempest-ubuntu-focal/9e1d5c6/logs/etc/placement/policy.yaml.txt
~~~
"rule:admin_api": "rule:system_reader_api"
"rule:admin_api": "rule:project_reader_api"
"rule:admin_api": "rule:system_or_project_reader"
~~~

[Takashi Kajinami (kajinamit)]

I tested clearing the policy.yaml file and it resolved the API error.

[Corey Bryant (corey.bryant)]

Takashi, Thank you for reporting this.

We do the following in the Ubuntu package source:

oslopolicy-sample-generator --config-file=etc/placement/policy-generator.conf
cp etc/placement/policy.yaml.sample etc/placement/policy.yaml

I wonder if we should drop the policy.yaml in recent releases due to the switch to policy in code.

[Takashi Kajinami (kajinamit)]

Although we can use empty policy files nowadays, the generated policy files should work in theory.

I suspect we need to backport the fix for https://bugs.launchpad.net/oslo.policy/+bug/1945336 .

I can reproduce the same content when I generate policy files using stable/xena or stable/yoga code.

As a short term workaround we can use empty files or modify the generated file to ensure all lines are commented out.

[Corey Bryant (corey.bryant)]

I compared the upstream generated policy.yaml from stable/xena with the package generated policy.yaml for stable/xena and they are the same.

The upstream policy.yaml can be generated with:
tox -e genpolicy # result is put in ./etc/placement/policy.yaml.sample

[Corey Bryant (corey.bryant)]

The policy.yaml documented here differs from what is generated with 'tox -e genpolicy':
https://docs.openstack.org/placement/latest/configuration/sample-policy.html

The version on the web has all policy commented out. This feels like it is not a package issue at this point.

Here is the diff: https://paste.ubuntu.com/p/BVPMzMNwkJ/

Pasting the diff here as well (formatting may not be great):

--- /tmp/policy.yaml 2022-08-29 16:54:11.582341611 +0000
+++ /etc/placement/policy.yaml 2021-10-07 15:21:07.000000000 +0000
@@ -16,16 +16,7 @@
 # Placement API policies are introducing new default roles with
 # scope_type capabilities. Old policies are deprecated and silently
 # going to be ignored in the placement 6.0.0 (Xena) release.
-# WARNING: A rule name change has been identified.
-# This may be an artifact of new rules being
-# included which require legacy fallback
-# rules to ensure proper policy behavior.
-# Alternatively, this may just be an alias.
-# Please evaluate on a case by case basis
-# keeping in mind the format for aliased
-# rules is:
-# "old_rule_name": "new_rule_name".
-# "rule:admin_api": "rule:system_admin_api"
+"rule:admin_api": "rule:system_admin_api"

 # Default rule for System level read only APIs.
 #"system_reader_api": "role:reader and system_scope:all"
@@ -36,16 +27,7 @@
 # Placement API policies are introducing new default roles with
 # scope_type capabilities. Old policies are deprecated and silently
 # going to be ignored in the placement 6.0.0 (Xena) release.
-# WARNING: A rule name change has been identified.
-# This may be an artifact of new rules being
-# included which require legacy fallback
-# rules to ensure proper policy behavior.
-# Alternatively, this may just be an alias.
-# Please evaluate on a case by case basis
-# keeping in mind the format for aliased
-# rules is:
-# "old_rule_name": "new_rule_name".
-# "rule:admin_api": "rule:system_reader_api"
+"rule:admin_api": "rule:system_reader_api"

 # Default rule for Project level read only APIs.
 #"project_reader_api": "role:reader and project_id:%(project_id)s"
@@ -56,16 +38,7 @@
 # Placement API policies are introducing new default roles with
 # scope_type capabilities. Old policies are deprecated and silently
 # going to be ignored in the placement 6.0.0 (Xena) release.
-# WARNING: A rule name change has been identified.
-# This may be an artifact of new rules being
-# included which require legacy fallback
-# rules to ensure proper policy behavior.
-# Alternatively, this may just be an alias.
-# Please evaluate on a case by case basis
-# keeping in mind the format for aliased
-# rules is:
-# "old_rule_name": "new_rule_name".
-# "rule:admin_api": "rule:project_reader_api"
+"rule:admin_api": "rule:project_reader_api"

 # Default rule for System+Project read only APIs.
 #"system_or_project_reader": "rule:system_reader_api or rule:project_reader_api"
@@ -77,16 +50,7 @@
 # Placement API policies are introducing new default roles with
 # scope_ty...

[Corey Bryant (corey.bryant)]

I'm going to triage this as invalid at this point since the package is generating the policy.yaml based on what is documented by upstream placement. If you have new information please share it on this bug and move the bug back to New status.