nova-api is crashed during creating instance

[Marian Horban]

If nova-api started by Apache(devstack config NOVA_USE_MOD_WSGI=True) it is crashed during creating instance with stack trace:

e81a26529b4ae2 and project 8a88d9cf7ff046a9a8b55e90466dec12. Resources: ['instances', 'ram', 'cores'] _get_quotas /opt/stack/nova/nova/quota.py:401
2015-10-16 11:54:13.486 DEBUG nova.quota [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] Quotas for project 8a88d9cf7ff046a9a8b55e9
0466dec12 and user 97caf8fda3ef497a8de81a26529b4ae2 after resource sync: {'instances': 10, 'ram': 51200, 'cores': 20} reserve /opt/stack/nova/nova/quota.py:573
2015-10-16 11:54:13.487 ERROR nova.api.openstack.extensions [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] Unexpected exception in
 API method
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/extensions.py", line 478, in wrapped
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/validation/__init__.py", line 73, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/api/openstack/compute/servers.py", line 602, in create
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions **create_kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/hooks.py", line 149, in inner
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions rv = f(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1569, in create
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions check_server_group_quota=check_server_group_quota)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 1192, in _create_instance
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions instance_group, check_server_group_quota)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 994, in _provision_instances
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions context, instance_type, min_count, max_count)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/compute/api.py", line 390, in _check_num_instances_quota
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions project_id=project_id, user_id=user_id)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 213, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions return fn(self, *args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/objects/quotas.py", line 92, in reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions **deltas)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/quota.py", line 1343, in reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions user_id=user_id)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/quota.py", line 583, in reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions project_id=project_id, user_id=user_id)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/api.py", line 1119, in quota_reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions project_id=project_id, user_id=user_id)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 204, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 147, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions ectxt.value = e.inner_exc
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in __exit__
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions six.reraise(self.type_, self.value, self.tb)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 137, in wrapper
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 3538, in quota_reserve
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions elevated = context.elevated()
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/opt/stack/nova/nova/context.py", line 200, in elevated
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions y = _reconstruct(x, rv, 1, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions state = deepcopy(state, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions y = copier(x, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions y[deepcopy(key, memo)] = deepcopy(value, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions y = _reconstruct(x, rv, 1, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 334, in _reconstruct
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions state = deepcopy(state, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 163, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions y = copier(x, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 257, in _deepcopy_dict
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions y[deepcopy(key, memo)] = deepcopy(value, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions y = _reconstruct(x, rv, 1, memo)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions y = callable(*args)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/copy_reg.py", line 93, in __newobj__
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions return cls.__new__(cls, *args)
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions TypeError: object.__new__(thread.lock) is not safe, use thread.lock.__new__()
2015-10-16 11:54:13.487 779 ERROR nova.api.openstack.extensions
2015-10-16 11:54:13.488 INFO nova.api.openstack.wsgi [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'>
2015-10-16 11:54:13.489 DEBUG nova.api.openstack.wsgi [req-eb2e7b74-e2b2-4248-86ab-9b672f447a06 tempest-ServersAdminTestJSON-1309219026 tempest-ServersAdminTestJSON-633645645] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<type 'exceptions.TypeError'> __call__ /opt/stack/nova/nova/api/openstack/wsgi.py:1175

[Marian Horban (mhorban)]

During work with quotas nova-api calls method

    nova.context.RequestContext.elevated()

Implementation contains code
    copy.deepcopy(self)

Because RequestContext object contains threading.Lock() object inside RequestContext.user_auth_plugin._lock object it could not be copied:

>>> import threading
>>> import copy
>>> x = threading.Lock()
>>> y = copy.deepcopy(x)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/copy.py", line 190, in deepcopy
    y = _reconstruct(x, rv, 1, memo)
  File "/usr/lib/python2.7/copy.py", line 329, in _reconstruct
    y = callable(*args)
  File "/usr/lib/python2.7/copy_reg.py", line 93, in __newobj__
    return cls.__new__(cls, *args)
TypeError: object.__new__(thread.lock) is not safe, use thread.lock.__new__()

This code worked well with NOVA_USE_MOD_WSGI=False because in that case nova-api uses eventlet server and calls monkey_patch(thread=True). After monkey patching threading.Lock() becomes copyable:

>>> import eventlet
>>> import threading
>>> import copy
>>> eventlet.monkey_patch(thread=True)
>>> x = threading.Lock()
>>> y = copy.deepcopy(x)
>>> y
<Semaphore at 0x7fa846804fd0 c=1 _w[0]>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/246521

[Boris Bobrov (bbobrov)]

If the bug is in the auth plugin, keystoneauth is affected too