Improve performance of community tag portlet

[Paul Everitt]

"""
I found your performance bottle neck. Not sure what to do about it yet, but it's the community tags portlet in the right sidebar of the community layout. well the call to tags.getFrequency() is taking 40-50 seconds on my local box. burns through a lot of CPU. possibly CPU bound rather than IO bound, but not positive. but would explain why ssd doesn't help.

looks like a possible O(n^2). the tagging stuff doesn't really contain an index for community. so if we do something like what we're doing in the community tags portlet, where we go:

  tags.getTags(community=community)

It marshalls every tag in the db and checks the community. It's probably only fixable with some kind of index on community.

[Chris Rossi (chris-archimedeanco)]

FWIW, the slow bit isn't O(n^2), it's just O(n) where n is fairly large.

[Paul Everitt (paul-agendaless)]

Handing some things over to Tres.

[Tres Seaver (tseaver)]

After getting the 'karl.tagging' test coverage to 100%, I have checked in
an optimization of getFrequency, avoiding intermediate calls to
'getTags' and 'getTagBojects' and the unnecessary creation of unused
intermediary data structures.

It would be useful if somebody with better knowledge of the current
deployment enviroment could test any speedup using production or
near-production data.

[Chris Rossi (chris-archimedeanco)]

I'm still seeing about 40+ seconds for this with OSI data. Although avoiding unnecessary marshalling of the tag objects is obviously a good thing, I think our fundamental problem is this code is marshalling every single tag object in the site:

return set([x for x in result
                            if self._tagid_to_obj[x].community == community])

I'm not sure I see any way of fixing this without adding a _community_to_tagids structure analogous to the other *_to_tagids structures already present for item, user, and name.

[Paul Everitt (paul-agendaless)]

I'll move it back to "in progress" and we'll decide whether to continue investigating it or not. Perhaps we'll throw the profiler at it.

[Tres Seaver (tseaver)]

The "slow load" issue is a cache locality problem. The following
computation:

 >>> [y for x, y in root.tags._tagid_to_obj.items()
 ... if y.community == 'nonesuch']
 []

takes many seconds (likely the 30-40 originally reported) on a cold ZODB
cache, but is nearly instantaneous on a hot cache.

Fixing this correctly likely involves generating sample data for profiling..

Here is some data, drawn from production:

 >>> len(root.tags._tagid_to_obj)
 37666
 >>> len(root.tags._item_to_tagids)
 7435
 >>> len(root.tags._name_to_tagids)
 4353
 >>> len(root.tags._user_to_tagids)
 500
 >>> len(set([y.community for x, y in root.tags._tagid_to_obj.items()]))
 384

[Paul Everitt (paul-agendaless)]

Chris did his idea on this and emailed the results of his branch to osi-dev. Tres, can you do a code review?

[Tres Seaver (tseaver)]

I have examined the diff::

 $ svn diff -c 5861 svn+ssh://<email address hidden>/home/osi/bfgsvn/karl/branches/community_tags_optimization

The main change looks good -- do we have enough test coverage for the cases
with / without communities to be confident that adding / using the index doesn't break
anything?

There seem to be some stray changes in the diff (comments in an older evolve script,
an apparently unrelated test), but they look harmless.

+1 for merge.

[Chris Rossi (chris-archimedeanco)]

Merged. Test coverage seems pretty good.

[JimPGlenn (jpglenn09)]

fixed