TypeError: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

[Dimitri John Ledkov]

When using /usr/bin/nova-api, running $ openstack availability zone list -> works fine.

If using the wsgi scripts, and running nova-api via e.g. uwsgi, the same client command fails as following:

2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions [req-184fd1f3-ae97-49d0-85dd-05ef08800238 0e56b818bc9c4eaea4b8d6a2f5da6227 906359c0c71749ceb27e46612e0419ce - - -] Unexpected exception in API method
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/availability_zone.py", line 115, in detail
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions return self._describe_availability_zones_verbose(context)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/availability_zone.py", line 61, in _describe_availability_zones_verbose
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions ctxt = context.elevated()
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/context.py", line 198, in elevated
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions context = copy.deepcopy(self)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions y = _reconstruct(x, rv, 1, memo)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions state = deepcopy(state, memo)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions y = copier(x, memo)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions y[deepcopy(key, memo)] = deepcopy(value, memo)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions y = _reconstruct(x, rv, 1, memo)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions state = deepcopy(state, memo)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions y = copier(x, memo)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions y[deepcopy(key, memo)] = deepcopy(value, memo)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions y = _reconstruct(x, rv, 1, memo)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions y = callable(*args)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy_reg.py", line 93, in __newobj__
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions return cls.__new__(cls, *args)
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions TypeError: object.__new__(thread.lock) is not safe, use thread.lock.__new__()
2015-10-16 16:58:20.720 18938 ERROR nova.api.openstack.extensions
2015-10-16 16:58:20.721 18938 INFO nova.api.openstack.wsgi [req-184fd1f3-ae97-49d0-85dd-05ef08800238 0e56b818bc9c4eaea4b8d6a2f5da6227 906359c0c71749ceb27e46612e0419ce - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>

Looks like a dejavu of https://bugs.launchpad.net/python-novaclient/+bug/1123561 but I am not certain.

This is with liberty final release (or so i believe at the moment).

[Davanum Srinivas (DIMS) (dims-v)]

Dimitri,

We don't see it in our tests running nova-api under Apache. what were the steps you used to run under uwsgi? There could be a code path that ends up adding more to the context which when deepcopy'ed causes issues in your case and not under apache.

-- Dims

[Victor Morales (electrocucaracha)]

It's weird we could only reproduce this issue using python-keystoneclient > 1.7.2 version.

[Dimitri John Ledkov (xnox)]

Horum, so I have deployed liberty/stable branch of devstack on fedora. And then killed nova-api, and run it via nginx+uwsgi, and I cannot replicate this bug.

I'm now wondering if we have mismatched package versions on our end, and/or stable branches had fixes since the tarball releases that we are missing.

[Sujitha (sujitha-neti)]

Hi

I'm able to reproduce the bug. I installed devstack on Ubuntu. Killed nova-api and changed configurations to run nova-api with apache2.

Command: $ openstack availability zone list

Output:
Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> (HTTP 500)

Attached the log file.

Note: Same issue with nova boot

[Dimitri John Ledkov (xnox)]

@ sujitha-neti

Could you please post version numbers of all packages involved from the openstack stuff? Things like nova, nova client, keystone, keystone client etc.

[Sujitha (sujitha-neti)]

@Dimitri

I'm working on devstack and below are the version numbers of related packages

Keystone client: 1.8.1
Keystone: 9.0.0
nova client: 2.32.0
nova: 13.0.0
apache2: 2.4.7

[Alberto Murillo (powerbsd-o)]

This bug was introduced at
https://github.com/openstack/nova/commit/06e2319806c618898071eba662d5bf9773be4d39

diff --git a/nova/context.py b/nova/context.py
index 9815bf3..e78636c 100644
--- a/nova/context.py
+++ b/nova/context.py
@@ -179,7 +179,7 @@ class RequestContext(object):

     def elevated(self, read_deleted=None, overwrite=False):
         """Return a version of this context with admin flag set."""
- context = copy.copy(self)
+ context = copy.deepcopy(self)
         context.is_admin = True

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/260615

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/260615
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=82457f2462621b6a9c653dce2baf38d0623e25ee
Submitter: Jenkins
Branch: master

commit 82457f2462621b6a9c653dce2baf38d0623e25ee
Author: Marian Horban <email address hidden>
Date: Mon Dec 7 07:30:11 2015 -0500

    Replace copy.deepcopy of RequestContext with copy.copy

    Instance of RequestContext contains many objects and some of them like
    mutexes could not be copied. Also a deepcopy of the entire
    RequestContext wastes CPU time.

    To avoid problems with deepcopy and avoid performance overhead this
    patch changes deepcopy of RequestContext to shallow copy and makes
    deepcopy of only the 'roles' member because of security issue
    LP #1386932.

    Closes-Bug: #1506958
    Related-Bug: #1386932
    Change-Id: I1e2c00e95e1c4bcd0ec7bf075458789d6fb06e99

[Thierry Carrez (ttx)]

This issue was fixed in the openstack/nova 13.0.0.0b2 development milestone.

[Matt Riedemann (mriedem)]

Removing kilo-backport-potential since I don't think we really supported running nova api with wsgi in kilo.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/288529

[Clayton O'Neill (clayton-oneill)]

Just ran into this problem with uwsgi on liberty. Would love to see this back port merged.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/288529
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3c2d75d60a1d11726905a9a3f8eb745b4e4ad4cd
Submitter: Jenkins
Branch: stable/liberty

commit 3c2d75d60a1d11726905a9a3f8eb745b4e4ad4cd
Author: Marian Horban <email address hidden>
Date: Mon Dec 7 07:30:11 2015 -0500

    Replace copy.deepcopy of RequestContext with copy.copy

    Instance of RequestContext contains many objects and some of them like
    mutexes could not be copied. Also a deepcopy of the entire
    RequestContext wastes CPU time.

    To avoid problems with deepcopy and avoid performance overhead this
    patch changes deepcopy of RequestContext to shallow copy and makes
    deepcopy of only the 'roles' member because of security issue
    LP #1386932.

    Closes-Bug: #1506958
    Related-Bug: #1386932
    Change-Id: I1e2c00e95e1c4bcd0ec7bf075458789d6fb06e99
    (cherry picked from commit 82457f2462621b6a9c653dce2baf38d0623e25ee)

[Davanum Srinivas (DIMS) (dims-v)]

This issue was fixed in the openstack/nova 12.0.3 release.