There is error:
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource Traceback (most recent call last):
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource result = method(request=request, **args)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 437, in create
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.force_reraise()
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource raise value
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.force_reraise()
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource raise value
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.force_reraise()
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource raise value
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 460, in _create
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource policy.init()
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/policy.py", line 94, in init
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource register_rules(_ENFORCER)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/policy.py", line 76, in register_rules
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource enforcer.register_defaults(itertools.chain(*policies))
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py", line 1119, in register_defaults
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.register_default(default)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py", line 1108, in register_default
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.registered_rules[default.name] = copy.deepcopy(default)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = _reconstruct(x, memo, *rv)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource state = deepcopy(state, memo)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = copier(x, memo)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y[deepcopy(key, memo)] = deepcopy(value, memo)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = _reconstruct(x, memo, *rv)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource state = deepcopy(state, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = copier(x, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y[deepcopy(key, memo)] = deepcopy(value, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = _reconstruct(x, memo, *rv)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource state = deepcopy(state, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = copier(x, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y[deepcopy(key, memo)] = deepcopy(value, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = _reconstruct(x, memo, *rv)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource state = deepcopy(state, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = copier(x, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y[deepcopy(key, memo)] = deepcopy(value, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 161, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource rv = reductor(4)
Feb 12 01:11:30.866920 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource TypeError: cannot pickle '_thread.RLock' object
Feb 12 01:11:30.866920 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource
while doing API request. E.g.:
2021-02-12 01:11:28.892252 | controller | ++ lib/neutron_plugins/services/l3:create_neutron_initial_network:176 : oscwrap --os-cloud devstack-admin --os-region RegionOne subnet pool create shared-default-subnetpool-v4 --default-prefix-length 26 --pool-prefix 10.1.0.0/20 --share --default -f value -c id
Logs: https://zuul.opendev.org/t/openstack/build/5e75395d4ef346de9f813ae678cd6fab/logs
Here is a python bug report that provides more details about the problems with pickling locks [0].
I can see how a particular lock implementation can be invoked by the deployment strategy, but it's unclear to me how an instance of RLock is being associated to the policy or registered defaults affected by the deep copy.
[0] https:/ /bugs.python. org/issue29168