Completion cache is not concurrency-safe

[James E. Blair]

The completion cache is not thread-safe if a Client is used by multiple threads

There is no locking around updating the cache so multiple threads may open, write to, and close the cache simultaneously. Since the cache file object is stored as an attribute on the Manager, they will use the same file handle if available, possibly resulting in writing to a closed file.

Sample tracebacks:

2013-08-19 14:17:45,887 ERROR nodepool.NodeLauncher: Exception launching node id: 123625:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nodepool/nodepool.py", line 156, in _run
    self.launchNode(session)
  File "/usr/local/lib/python2.7/dist-packages/nodepool/nodepool.py", line 176, in launchNode
    flavor = utils.get_flavor(self.client, self.image.min_ram)
  File "/usr/local/lib/python2.7/dist-packages/nodepool/nodeutils.py", line 83, in get_flavor
    flavors = [f for f in client.flavors.list() if f.ram >= min_ram]
  File "/usr/local/lib/python2.7/dist-packages/novaclient/v1_1/flavors.py", line 99, in list
    return self._list("/flavors%s%s" % (detail, query_string), "flavors")
  File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 78, in _list
    for res in data if res]
  File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 398, in __init__
    self.manager.write_to_completion_cache('human_id', human_id)
  File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 136, in write_to_completion_cache
    cache.write("%s\n" % val)
ValueError: I/O operation on closed file

2013-08-19 14:17:45,887 ERROR nodepool.NodeLauncher: Exception launching node id: 123626:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/nodepool/nodepool.py", line 156, in _run
    self.launchNode(session)
  File "/usr/local/lib/python2.7/dist-packages/nodepool/nodepool.py", line 176, in launchNode
    flavor = utils.get_flavor(self.client, self.image.min_ram)
  File "/usr/local/lib/python2.7/dist-packages/nodepool/nodeutils.py", line 83, in get_flavor
    flavors = [f for f in client.flavors.list() if f.ram >= min_ram]
  File "/usr/local/lib/python2.7/dist-packages/novaclient/v1_1/flavors.py", line 99, in list
    return self._list("/flavors%s%s" % (detail, query_string), "flavors")
  File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 78, in _list
    for res in data if res]
  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
    self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/novaclient/base.py", line 131, in completion_cache
    delattr(self, cache_attr)
AttributeError: _human_id_cache

[PWP (pwp333)]

This is severely affecting the speed to provision multiple vms as we can only use one thread now.

[Ryan Wilson (wryan)]

Here's a very simple change that fixes the cache thread safety issue.

[Ryan Wilson (wryan)]

In code review, Please take a look at https://review.openstack.org/#/c/51941/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/51941
Committed: http://github.com/openstack/python-novaclient/commit/c32056287704e4ce6e616bbbc37b7be98c44b218
Submitter: Jenkins
Branch: master

commit c32056287704e4ce6e616bbbc37b7be98c44b218
Author: Ryan Wilson <email address hidden>
Date: Tue Oct 15 06:09:43 2013 -0700

    Adds locking to completion caches

    Previously, completion caches were not thread-safe and multiple
    threads could not make calls to python-novaclient simultaneously.
    With locking, the completion caches are now thread-safe.

    Closes-Bug: #1213958
    Change-Id: I201c2ed0f1eb09b1c2a74de96119cbae9ed6f4b0

[melanie witt (melwitt)]

novaclient 2.16.0 released on 2/26/2014