Multiple problems with thread safe.

Bug #1412845 reported by Ivan
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Storm
New
Undecided
Unassigned

Bug Description

Hi, all.

I've an error on highload uwsgi.

Maybe initialize_globals() should lock threads.

There is single Store object for each thread, according docs:
https://storm.canonical.com/Manual#Working_with_multiple_threads

  File "./community/views/components/sharing.py", line 33, in process
    twitter_profile = store.get(StTwitterProfile, (context_id, site.owner_id))

  File "./.virtualenvs/local/lib/python2.7/site-packages/storm/store.py",
line 181, in get
    variable = column.variable_factory(value=variable)

RuntimeError: initialize_globals() failed the first time it was run

Revision history for this message
Ivan (ivzak) wrote :

The problem is stably reproduced on reload uwsgi.

Revision history for this message
Ivan (ivzak) wrote :

There are also multiple treadsafe bugs for global dicts and other global objects.

For example, create simple uwsgi application with 5 processes and 5 threads. Run ab util:

ab -n 100 -c 5 -C sessionid='123' https://myhost.loc/

And in process try reload uwsgi. Concurrent threads can reset global dicts and attributes.

Modification of global data should lock other threads.

The most often error is:

  File ".virtualenvs/local/lib/python2.7/site-packages/storm/properties.py", line 60, in __get__
    return obj_info.variables[column].get()

KeyError: <storm.properties.PropertyColumn object at 0x7fe6b141dad0>

description: updated
Revision history for this message
Ivan (ivzak) wrote :

The temporary solution to fix "RuntimeError: initialize_globals() failed the first time it was run" is execute on module level next code:

# Under imp.acquire_lock in django module loader
# see AppCache._populate()
# Fixed bug RuntimeError: initialize_globals() failed the first time it was run
# Create first Variable instance with lock threads.
from storm.variables import Variable
Variable()

Now working well.

Revision history for this message
Ivan (ivzak) wrote :

Too many bags for now on reload uwsgi, like this:

  File ".virtualenvs/local/lib/python2.7/site-packages/storm/properties.py", line 60, in __get__
    return obj_info.variables[column].get()

KeyError: <storm.properties.PropertyColumn object at 0x7fe6b141dad0>

Revision history for this message
Ivan (ivzak) wrote :

Also too many bugs on uwsgi reload like this:

File "./community/views/renderers.py", line 325, in <setcomp>
    has_twitter = {i.user_id for i in store.find(

  File ".virtualenvs/local/lib/python2.7/site-packages/storm/store.py", line 996, in __iter__
    yield self._load_objects(result, values)

  File ".virtualenvs/local/lib/python2.7/site-packages/storm/store.py", line 989, in _load_objects
    return self._find_spec.load_objects(self._store, result, values)

  File ".virtualenvs/local/lib/python2.7/site-packages/storm/store.py", line 1734, in load_objects
    values[values_start:values_end])

  File ".virtualenvs/local/lib/python2.7/site-packages/storm/store.py", line 714, in _load_object
    values, keep_defined=True)

  File ".virtualenvs/local/lib/python2.7/site-packages/storm/store.py", line 766, in _set_values
    variable = obj_info.variables[column]

KeyError: <storm.properties.PropertyColumn object at 0x7f2c98b35830>

Revision history for this message
Ivan (ivzak) wrote :

There is a code to reproduce bug:

from storm_ext import stores
from users.orm import TwitterProfile as StTwitterProfile

def t():
    tp = stores['default_slave'].get(StTwitterProfile, (0, 1))
    a = tp.provider_user_name

for i in range(100):
    threading.Thread(target=t).start()

It's thread-safe bug:

Exception in thread Thread-995:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "seeds/storm_threads.py", line 22, in t
    print tp.provider_user_name
  File "/mnt/local/home/zakrevskyi/.virtualenvs/rebelmouse/local/lib/python2.7/site-packages/storm/properties.py", line 60, in __get__
    return obj_info.variables[column].get()
KeyError: <storm.properties.PropertyColumn object at 0x7f0657ecef30>

Exception in thread Thread-998:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "seeds/storm_threads.py", line 22, in t
    print tp.provider_user_name
  File "/mnt/local/home/zakrevskyi/.virtualenvs/rebelmouse/local/lib/python2.7/site-packages/storm/properties.py", line 60, in __get__
    return obj_info.variables[column].get()
KeyError: <storm.properties.PropertyColumn object at 0x7f0657ecef30>

Exception in thread Thread-999:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "seeds/storm_threads.py", line 22, in t
    print tp.provider_user_name
  File "/mnt/local/home/zakrevskyi/.virtualenvs/rebelmouse/local/lib/python2.7/site-packages/storm/properties.py", line 60, in __get__
    return obj_info.variables[column].get()
KeyError: <storm.properties.PropertyColumn object at 0x7f0657ecef30>

Exception in thread Thread-1000:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "seeds/storm_threads.py", line 22, in t
    print tp.provider_user_name
  File "/mnt/local/home/zakrevskyi/.virtualenvs/rebelmouse/local/lib/python2.7/site-packages/storm/properties.py", line 60, in __get__
    return obj_info.variables[column].get()
KeyError: <storm.properties.PropertyColumn object at 0x7f0657ecef30>

Revision history for this message
Ivan (ivzak) wrote :
Download full text (3.4 KiB)

You can see, that <storm.properties.PropertyColumn object at 0x7fb4c1652050> exists in StTwitterProfile._storm_columns but not exists in tp.__storm_object_info__.variables

ipdb> StTwitterProfile._storm_columns
ipdb> {<storm.properties.Int object at 0x7fb4c1e99810>: <storm.properties.PropertyColumn object at 0x7fb4c1e839f0>,
<storm.properties.Unicode object at 0x7fb4c1e99b10>: <storm.properties.PropertyColumn object at 0x7fb4c1e83de0>,
<storm.properties.Int object at 0x7fb4c1e99a90>: <storm.properties.PropertyColumn object at 0x7fb4c1e83c90>,
<storm.properties.Unicode object at 0x7fb4c1e998d0>: <storm.properties.PropertyColumn object at 0x7fb4c1652050>,
<storm.properties.Int object at 0x7fb4c1e997d0>: <storm.properties.PropertyColumn object at 0x7fb4c1e83d00>,
<storm.properties.Unicode object at 0x7fb4c1e99910>: <storm.properties.PropertyColumn object at 0x7fb4c1e83bb0>,
<storm.properties.Int object at 0x7fb4c1e99950>: <storm.properties.PropertyColumn object at 0x7fb4c1e83f30>,
<storm.properties.Int object at 0x7fb4c1e99890>: <storm.properties.PropertyColumn object at 0x7fb4c1e83e50>,
<storm.properties.DateTime object at 0x7fb4c1e99990>: <storm.properties.PropertyColumn object at 0x7fb4c1e83c20>,
<storm.properties.Unicode object at 0x7fb4c1e99a10>: <storm.properties.PropertyColumn object at 0x7fb4c1e83fa0>,
<storm.properties.DateTime object at 0x7fb4c1e999d0>: <storm.properties.PropertyColumn object at 0x7fb4c1e83d70>}

ipdb> StTwitterProfile.__storm_class_info__.columns
ipdb> (<storm.properties.PropertyColumn object at 0x7fb4c1e83bb0>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83c20>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83c90>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83d00>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83d70>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83de0>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83e50>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83ec0>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83f30>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83fa0>,
<storm.properties.PropertyColumn object at 0x7fb4c1e839f0>)

ipdb> tp.__storm_object_info__.variables
ipdb> {<storm.properties.PropertyColumn object at 0x7fb4c1e83c20>: <storm.variables.DateTimeVariable object at 0x7fb47012d5d0>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83e50>: <storm.variables.IntVariable object at 0x7fb47013e758>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83c90>: <storm.variables.IntVariable object at 0x7fb47013e5f0>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83ec0>: <storm.variables.UnicodeVariable object at 0x7fb47013e7d0>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83d00>: <storm.variables.IntVariable object at 0x7fb47013e668>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83f30>: <storm.variables.IntVariable object at 0x7fb47013e848>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83d70>: <storm.variables.DateTimeVariable object at 0x7fb47012d650>,
<storm.properties.PropertyColumn object at 0x7fb4c1e83fa0>: <storm.variables.UnicodeVariable object at 0x7fb47013e8c0>,
<storm.properties.PropertyColumn object a...

Read more...

Revision history for this message
Ivan (ivzak) wrote :

Property._get_column() should lock threads during creating column, when column does not exists in cls.__dict__["_storm_columns"]

Ivan (ivzak)
summary: - initialize_globals() is not thread safe
+ Multiple problems with thread safe.
Revision history for this message
Ivan (ivzak) wrote :

Maybe class Storm as parent for models can resolve problem KeyError: http://bazaar.launchpad.net/~storm/storm/trunk/view/469/storm/properties.py#L303

Revision history for this message
Ivan (ivzak) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.