DecoratedResultSet eagerly fetches all results

[Ian Booth]

The DecoratedResultSet class implements the __iter__ method using yield as expected, but the first line of the method does this:

results = list(self.result_set.__iter__(*args, **kwargs))

This of course eagerly loads the entire underlying set of results and is a definite performance killer for larger result sets and/or for where there are multiple concurrent queries using DecoratedResultSet.

The __iter__ implementation really must be changed so as not to eagerly load all records.

[Robert Collins (lifeless)]

So, I spent some time looking/thinking about this previously; there are multiple reasons why this isn't needed/is a problem to implement.

Firstly, our *database* layer doesn't iterate: queries are satisfied entirely into a buffer which pgsql then reads from.

Secondly, the code in DRS would be much more complex - because the pre_iter_hook would have to still fire when appropriate.

Thirdly, multiple layers within storm realise things eagerly too - to actually do this you need to go very deep : and there are correctness/race conditions you'll create by doing that.

Lastly nothing in our stack /should/ depend on lazy evaluation : templates in particular never half-render.

Obviously slicing is different - but we slice before we query, not after.