neutron

Neutron run with uwsgi is broken with oslo_policy 3.6.2

Bug #1915494 reported by Slawek Kaplonski
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Critical
Slawek Kaplonski
neutron wallaby-3

Bug Description

There is error:

Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource Traceback (most recent call last):
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource result = method(request=request, **args)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 437, in create
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.force_reraise()
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource raise value
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper
Feb 12 01:11:30.857281 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.force_reraise()
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource raise value
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource return f(*args, **kwargs)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource LOG.debug("Retry wrapper got retriable exception: %s", e)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.force_reraise()
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource raise value
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs)
Feb 12 01:11:30.858866 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 460, in _create
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource policy.init()
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/policy.py", line 94, in init
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource register_rules(_ENFORCER)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/policy.py", line 76, in register_rules
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource enforcer.register_defaults(itertools.chain(*policies))
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py", line 1119, in register_defaults
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.register_default(default)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py", line 1108, in register_default
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource self.registered_rules[default.name] = copy.deepcopy(default)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = _reconstruct(x, memo, *rv)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource state = deepcopy(state, memo)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = copier(x, memo)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y[deepcopy(key, memo)] = deepcopy(value, memo)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = _reconstruct(x, memo, *rv)
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
Feb 12 01:11:30.861452 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource state = deepcopy(state, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = copier(x, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y[deepcopy(key, memo)] = deepcopy(value, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = _reconstruct(x, memo, *rv)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource state = deepcopy(state, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = copier(x, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y[deepcopy(key, memo)] = deepcopy(value, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 172, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = _reconstruct(x, memo, *rv)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 270, in _reconstruct
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource state = deepcopy(state, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 146, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y = copier(x, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 230, in _deepcopy_dict
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource y[deepcopy(key, memo)] = deepcopy(value, memo)
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource File "/usr/lib/python3.8/copy.py", line 161, in deepcopy
Feb 12 01:11:30.864190 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource rv = reductor(4)
Feb 12 01:11:30.866920 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource TypeError: cannot pickle '_thread.RLock' object
Feb 12 01:11:30.866920 ubuntu-focal-rax-iad-0022985630 <email address hidden>[68621]: ERROR neutron.api.v2.resource

while doing API request. E.g.:

2021-02-12 01:11:28.892252 | controller | ++ lib/neutron_plugins/services/l3:create_neutron_initial_network:176 : oscwrap --os-cloud devstack-admin --os-region RegionOne subnet pool create shared-default-subnetpool-v4 --default-prefix-length 26 --pool-prefix 10.1.0.0/20 --share --default -f value -c id

Logs: https://zuul.opendev.org/t/openstack/build/5e75395d4ef346de9f813ae678cd6fab/logs

Tags: api uwsgi
Revision history for this message
Lance Bragstad (lbragstad) wrote :

Here is a python bug report that provides more details about the problems with pickling locks [0].

I can see how a particular lock implementation can be invoked by the deployment strategy, but it's unclear to me how an instance of RLock is being associated to the policy or registered defaults affected by the deep copy.

[0] https://bugs.python.org/issue29168

Revision history for this message
Brian Haley (brian-haley) wrote :

I can confirm this also is breaking the ovn-octavia-provider unit test jobs, lots of 'pickle' errors in there. I think we also need to blacklist 3.6.2 in requirements as is done with 3.6.1 since I can't seem to do it locally without a requirements mismatch failure.

Jakub Libosvar (libosvar)
Changed in neutron:
milestone: none → wallaby-3
Bernard Cafarelli (bcafarel)
tags: added: neutron-proactive-backport-potential
Slawek Kaplonski (slaweq)
Changed in neutron:
status: Confirmed → Fix Released
tags: removed: neutron-proactive-backport-potential
Revision history for this message
Lajos Katona (lajos-katona) wrote :

The patch was merged: https://review.opendev.org/c/openstack/neutron/+/775357

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 18.0.0.0rc1

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.