Slim down Variable when cextensions are used
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Storm |
Won't Fix
|
Undecided
|
Unassigned |
Bug Description
The cextensions work has helped improve speed of Storm a lot, but has left memory usage almost the same as before (well, the same if we use slots in the Python version).
Given that Variable objects will be some of the most common objects in a large application, it'd make sense to try and reduce the size of those objects. Here are a few ideas:
* _allow_none is stored as a PyObject pointer, while it only needs a single bit. On AMD64, this is almost 8 bytes wasted.
* if we add a flag to say whether the value is lazy or not, we can collapse _value and _lazy_value to a single pointer.
* can we determine _validator_
* can we determine event from _validator_
So this will save a few words of memory per Variable instance. When you multiply that by the number of Storm objects and number of columns on each of those objects, this adds up.
Changed in storm: | |
status: | New → Won't Fix |
From discussion on IRC:
20:24 <jamesh> niemeyer: another way to go would be a three-level hierarchy attribute and column object_ factory and event change per instance
20:24 <jamesh> e.g. IntVariable -> IntVariable with validator -> instance
20:25 <jamesh> converting some of the variable_factory stuff into classes rather than partial function application
20:29 <niemeyer> jamesh: Interesting. I wonder how much that'd save.. I remember that building the validator stuff required collecting data from a few different places, which is why we ended up like this
20:30 <niemeyer> jamesh: E.g. the partial function is already unique per class/variable, and the object itself can't be made into the IntVariable class with validator
20:30 <jamesh> niemeyer: well, in the existing VariableObject struct, I see _allow_none, _validator, _validator_
20:31 <niemeyer> jamesh: Ok
20:31 <jamesh> niemeyer: only _value, _lazy_value, _checkpoint_state, _validator_
20:31 <jamesh> so that could be a reduction of 12/24 bytes per Variable
20:33 <jamesh> the only variable classes that add extra attributes are DateTimeVariable (_tzinfo) and EnumVariable (_get_map, _set_map)
20:33 <jamesh> both cases where we have per (class,column) data
20:34 <niemeyer> jamesh: Right, makes sense