Investigate accelerating / caching tagcloud computation

[Tres Seaver]

The tag cloud views are infrequently accessed, but when they *are* invoked, they tend to evict "important" objects from the ZODB caches.

Figure out the feasibility of computing the underlying cloud less frequently (e.g., nightly, in an out-of-appserver process) and serve only using the cached counts.

[Chris Rossi (chris-archimedeanco)]

I've been staring at this. Essentially, the tag cloud is just a mapping of tag name to tag count. I think what I would probably do is add a data structure to Tags that stores mappings of tag name to tag count that are maintained live as tags are updated. I'd have one structure that kept the global counts and another that kept counts for each community. This gets us over having to have some kind of nightly count process, but still makes the information we need available and precomputed.

[Paul Everitt (paul-agendaless)]

I'll admit that the idea of keeping some out-of-band process up-to-date has given me the heebie jeebies. Those things are constantly sore spots.

--Paul

On Aug 20, 2014, at 1:39 PM, Chris Rossi <email address hidden> wrote:

> I've been staring at this. Essentially, the tag cloud is just a mapping
> of tag name to tag count. I think what I would probably do is add a
> data structure to Tags that stores mappings of tag name to tag count
> that are maintained live as tags are updated. I'd have one structure
> that kept the global counts and another that kept counts for each
> community. This gets us over having to have some kind of nightly count
> process, but still makes the information we need available and
> precomputed.
>
> --
> You received this bug notification because you are subscribed to KARL3.
> https://bugs.launchpad.net/bugs/1302128
>
> Title:
> Investigate accelerating / caching tagcloud computation
>
> Status in KARL3:
> In Progress
>
> Bug description:
> The tag cloud views are infrequently accessed, but when they *are*
> invoked, they tend to evict "important" objects from the ZODB caches.
>
> Figure out the feasibility of computing the underlying cloud less
> frequently (e.g., nightly, in an out-of-appserver process) and serve
> only using the cached counts.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/1302128/+subscriptions

[Chris Rossi (chris-archimedeanco)]

Should I take that as a "make it so" or wait for further instructions?
 Just to play devil's advocate, it is kind of a lot of trouble to go to
maintain such a seldom used feature. It could also be a candidate for
removal, like the tag portlets.

Chris

On Wed, Aug 20, 2014 at 1:52 PM, Paul Everitt <email address hidden> wrote:

> I'll admit that the idea of keeping some out-of-band process up-to-date
> has given me the heebie jeebies. Those things are constantly sore spots.
>
>

[Paul Everitt (paul-agendaless)]

Unfortunately the tag cloud is something that would be noticed if it disappeared. I will leave it your judgement to decide on nightly computing or do your idea.

--Paul

On Aug 20, 2014, at 3:12 PM, Chris Rossi <email address hidden> wrote:

> Should I take that as a "make it so" or wait for further instructions?
> Just to play devil's advocate, it is kind of a lot of trouble to go to
> maintain such a seldom used feature. It could also be a candidate for
> removal, like the tag portlets.
>
> Chris
>
> On Wed, Aug 20, 2014 at 1:52 PM, Paul Everitt <email address hidden>
> wrote:
>
>> I'll admit that the idea of keeping some out-of-band process up-to-date
>> has given me the heebie jeebies. Those things are constantly sore spots.
>>
>>
>
> --
> You received this bug notification because you are subscribed to KARL3.
> https://bugs.launchpad.net/bugs/1302128
>
> Title:
> Investigate accelerating / caching tagcloud computation
>
> Status in KARL3:
> In Progress
>
> Bug description:
> The tag cloud views are infrequently accessed, but when they *are*
> invoked, they tend to evict "important" objects from the ZODB caches.
>
> Figure out the feasibility of computing the underlying cloud less
> frequently (e.g., nightly, in an out-of-appserver process) and serve
> only using the cached counts.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/1302128/+subscriptions

[Chris Rossi (chris-archimedeanco)]

Ok, so going ahead and doing something is in the scope of this ticket,
right?

On Wed, Aug 20, 2014 at 4:17 PM, Paul Everitt <email address hidden> wrote:

> Unfortunately the tag cloud is something that would be noticed if it
> disappeared. I will leave it your judgement to decide on nightly computing
> or do your idea.
>
> --Paul
>
> On Aug 20, 2014, at 3:12 PM, Chris Rossi <email address hidden>
> wrote:
>
> > Should I take that as a "make it so" or wait for further instructions?
> > Just to play devil's advocate, it is kind of a lot of trouble to go to
> > maintain such a seldom used feature. It could also be a candidate for
> > removal, like the tag portlets.
> >
> > Chris
> >
> > On Wed, Aug 20, 2014 at 1:52 PM, Paul Everitt <email address hidden>
> > wrote:
> >
> >> I'll admit that the idea of keeping some out-of-band process up-to-date
> >> has given me the heebie jeebies. Those things are constantly sore spots.
> >>
> >>
> >
> > --
> > You received this bug notification because you are subscribed to KARL3.
> > https://bugs.launchpad.net/bugs/1302128
> >
> > Title:
> > Investigate accelerating / caching tagcloud computation
> >
> > Status in KARL3:
> > In Progress
> >
> > Bug description:
> > The tag cloud views are infrequently accessed, but when they *are*
> > invoked, they tend to evict "important" objects from the ZODB caches.
> >
> > Figure out the feasibility of computing the underlying cloud less
> > frequently (e.g., nightly, in an out-of-appserver process) and serve
> > only using the cached counts.
> >
> > To manage notifications about this bug go to:
> > https://bugs.launchpad.net/karl3/+bug/1302128/+subscriptions
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/1302128
>
> Title:
> Investigate accelerating / caching tagcloud computation
>
> Status in KARL3:
> In Progress
>
> Bug description:
> The tag cloud views are infrequently accessed, but when they *are*
> invoked, they tend to evict "important" objects from the ZODB caches.
>
> Figure out the feasibility of computing the underlying cloud less
> frequently (e.g., nightly, in an out-of-appserver process) and serve
> only using the cached counts.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/karl3/+bug/1302128/+subscriptions
>

[Paul Everitt (paul-agendaless)]

Yep. We just can't do like the tag portlet and simply remove the page. ;)

--Paul

On Aug 20, 2014, at 4:32 PM, Chris Rossi <email address hidden> wrote:

> Ok, so going ahead and doing something is in the scope of this ticket,
> right?
>
>
> On Wed, Aug 20, 2014 at 4:17 PM, Paul Everitt <email address hidden>
> wrote:
>
>> Unfortunately the tag cloud is something that would be noticed if it
>> disappeared. I will leave it your judgement to decide on nightly computing
>> or do your idea.
>>
>> --Paul
>>
>> On Aug 20, 2014, at 3:12 PM, Chris Rossi <email address hidden>
>> wrote:
>>
>>> Should I take that as a "make it so" or wait for further instructions?
>>> Just to play devil's advocate, it is kind of a lot of trouble to go to
>>> maintain such a seldom used feature. It could also be a candidate for
>>> removal, like the tag portlets.
>>>
>>> Chris
>>>
>>> On Wed, Aug 20, 2014 at 1:52 PM, Paul Everitt <email address hidden>
>>> wrote:
>>>
>>>> I'll admit that the idea of keeping some out-of-band process up-to-date
>>>> has given me the heebie jeebies. Those things are constantly sore spots.
>>>>
>>>>
>>>
>>> --
>>> You received this bug notification because you are subscribed to KARL3.
>>> https://bugs.launchpad.net/bugs/1302128
>>>
>>> Title:
>>> Investigate accelerating / caching tagcloud computation
>>>
>>> Status in KARL3:
>>> In Progress
>>>
>>> Bug description:
>>> The tag cloud views are infrequently accessed, but when they *are*
>>> invoked, they tend to evict "important" objects from the ZODB caches.
>>>
>>> Figure out the feasibility of computing the underlying cloud less
>>> frequently (e.g., nightly, in an out-of-appserver process) and serve
>>> only using the cached counts.
>>>
>>> To manage notifications about this bug go to:
>>> https://bugs.launchpad.net/karl3/+bug/1302128/+subscriptions
>>
>> --
>> You received this bug notification because you are a bug assignee.
>> https://bugs.launchpad.net/bugs/1302128
>>
>> Title:
>> Investigate accelerating / caching tagcloud computation
>>
>> Status in KARL3:
>> In Progress
>>
>> Bug description:
>> The tag cloud views are infrequently accessed, but when they *are*
>> invoked, they tend to evict "important" objects from the ZODB caches.
>>
>> Figure out the feasibility of computing the underlying cloud less
>> frequently (e.g., nightly, in an out-of-appserver process) and serve
>> only using the cached counts.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/karl3/+bug/1302128/+subscriptions
>>
>
> --
> You received this bug notification because you are subscribed to KARL3.
> https://bugs.launchpad.net/bugs/1302128
>
> Title:
> Investigate accelerating / caching tagcloud computation
>
> Status in KARL3:
> In Progress
>
> Bug description:
> The tag cloud views are infrequently accessed, but when they *are*
> invoked, they tend to evict "important" objects from the ZODB caches.
>
> Figure out the feasibility of computing the underlying cloud less
> frequently (e.g., nightly, in an out-of-appserver process) and serve
> only using the cached counts.
>
> To manage notifications about this bug go to:
> http...

[Chris Rossi (chris-archimedeanco)]

Fix commited on branch: precompute-tag-clouds-lp1302128

[Paul Everitt (paul-agendaless)]

After merging and doing:

$ bin/karlstaging evolve

...my local sandbox gives the following AttributeError:

File '/Users/paul/projects/karl/dev-buildout/src/karl/karl/views/tags.py', line 324 in tag_cloud_view
  cloud = [{'name': x[0], 'count': x[1]} for x in tags.getCloud()]
File '/Users/paul/projects/karl/dev-buildout/src/karl/karl/tagging/__init__.py', line 169 in getCloud
  d = self._global_cloud
AttributeError: 'Tags' object has no attribute '_global_cloud'

URL: http://localhost:6543/pg/tagcloud.html
File '/Users/paul/projects/karl/dev-buildout/eggs/Paste-1.7.5.1-py2.7.egg/paste/evalexception/middleware.py', line 306 in respond
  app_iter = self.application(environ, detect_start_response)
File '/Users/paul/projects/karl/dev-buildout/src/karl/karl/timeit.py', line 48 in __call__
  resp = req.get_response(self.app)
File '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py', line 1086 in get_response
  application, catch_exc_info=False)
File '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py', line 1055 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/paul/projects/karl/dev-buildout/eggs/repoze.browserid-0.3-py2.7.egg/repoze/browserid/middleware.py', line 127 in __call__
  return self.app(environ, start_response)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py', line 176 in __call__
  response = self.handle_request(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/tweens.py', line 17 in excview_tween
  response = handler(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py', line 153 in handle_request
  response = view_callable(context, request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py', line 319 in viewresult_to_response
  result = view(context, request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py', line 402 in _requestonly_view
  response = view(request)
File '/Users/paul/projects/karl/dev-buildout/src/karlserve/karlserve/application.py', line 135 in site_dispatch
  return request.get_response(instance.pipeline())
File '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py', line 1086 in get_response
  application, catch_exc_info=False)
File '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py', line 1055 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py', line 176 in __call__
  response = self.handle_request(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/perfmetrics-2.0-py2.7.egg/perfmetrics/__init__.py', line 270 in handle
  return handler(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/slowlog-0.9-py2.7.egg/slowlog/tween.py', line 38 in __call__
  return self.handler(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/tweens.py', line 17 in excview_tween
  re...

[Paul Everitt (paul-agendaless)]

I will go ahead and re-open this just to be safe, knowing it might be my Friday stupidity.