Reduce the objects loaded for tags portlet

[Paul Everitt]

Shane's analysis showed that loading a wiki page loads thousands of tag objects, due to the portlet.

First choice is to get rid of the portlet. But if that isn't allowed, we need to find some remediation techniques:

- Is it a bad query? Missing index? Unfortunately the tags datastructure was extremely difficult to concoct.

- If not, should we make it an async request....load the page then ask for the JSON data for that portlet?

The tag portlet in question has the following tags.

570 mexico
513 colombia
475 venezuela
412 brazil
330 peru

The tag listing shows what might be 200 tags for that community.

[Paul Everitt (paul-agendaless)]

Per conversation with Nat, we can just remove the tags portlet in the places it is used. Yay!

[Paul Everitt (paul-agendaless)]

Since Nat said we can just remove it, and since the impact will likely be large, I'm moving this one up.

[Chris Rossi (chris-archimedeanco)]

Fix commited on branch: remove-tags-portlet-lp1357406

[Paul Everitt (paul-agendaless)]

This also had a couple of similar tracebacks, include _community_cloud. Both pasted below, occurred when I tagged something.

URL: http://localhost:6543/pg/tagcloud.html
File '/Users/paul/projects/karl/dev-buildout/eggs/Paste-1.7.5.1-py2.7.egg/paste/evalexception/middleware.py', line 306 in respond
  app_iter = self.application(environ, detect_start_response)
File '/Users/paul/projects/karl/dev-buildout/src/karl/karl/timeit.py', line 48 in __call__
  resp = req.get_response(self.app)
File '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py', line 1086 in get_response
  application, catch_exc_info=False)
File '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py', line 1055 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/paul/projects/karl/dev-buildout/eggs/repoze.browserid-0.3-py2.7.egg/repoze/browserid/middleware.py', line 127 in __call__
  return self.app(environ, start_response)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py', line 176 in __call__
  response = self.handle_request(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/tweens.py', line 17 in excview_tween
  response = handler(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py', line 153 in handle_request
  response = view_callable(context, request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py', line 319 in viewresult_to_response
  result = view(context, request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py', line 402 in _requestonly_view
  response = view(request)
File '/Users/paul/projects/karl/dev-buildout/src/karlserve/karlserve/application.py', line 135 in site_dispatch
  return request.get_response(instance.pipeline())
File '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py', line 1086 in get_response
  application, catch_exc_info=False)
File '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py', line 1055 in call_application
  app_iter = application(self.environ, start_response)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py', line 176 in __call__
  response = self.handle_request(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/perfmetrics-2.0-py2.7.egg/perfmetrics/__init__.py', line 270 in handle
  return handler(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/slowlog-0.9-py2.7.egg/slowlog/tween.py', line 38 in __call__
  return self.handler(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/tweens.py', line 17 in excview_tween
  response = handler(request)
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py', line 107 in tm_tween
  return response
File '/Users/paul/projects/karl/dev-buildout/eggs/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py', line 75 in __exit__
  return self._retry_or_raise(t, v, tb)
File '/Users/paul/...

[Chris Rossi (chris-archimedeanco)]

There's an evolve step for this one. Did you use the osideploy script?

Chris

On Fri, Aug 22, 2014 at 2:34 PM, Paul Everitt <email address hidden> wrote:

> This also had a couple of similar tracebacks, include _community_cloud.
> Both pasted below, occurred when I tagged something.
>
> URL: http://localhost:6543/pg/tagcloud.html
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/Paste-1.7.5.1-py2.7.egg/paste/evalexception/middleware.py',
> line 306 in respond
> app_iter = self.application(environ, detect_start_response)
> File '/Users/paul/projects/karl/dev-buildout/src/karl/karl/timeit.py',
> line 48 in __call__
> resp = req.get_response(self.app)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py',
> line 1086 in get_response
> application, catch_exc_info=False)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py',
> line 1055 in call_application
> app_iter = application(self.environ, start_response)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/repoze.browserid-0.3-py2.7.egg/repoze/browserid/middleware.py',
> line 127 in __call__
> return self.app(environ, start_response)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py',
> line 176 in __call__
> response = self.handle_request(request)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/tweens.py',
> line 17 in excview_tween
> response = handler(request)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py',
> line 153 in handle_request
> response = view_callable(context, request)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py',
> line 319 in viewresult_to_response
> result = view(context, request)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py',
> line 402 in _requestonly_view
> response = view(request)
> File
> '/Users/paul/projects/karl/dev-buildout/src/karlserve/karlserve/application.py',
> line 135 in site_dispatch
> return request.get_response(instance.pipeline())
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py',
> line 1086 in get_response
> application, catch_exc_info=False)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py',
> line 1055 in call_application
> app_iter = application(self.environ, start_response)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py',
> line 176 in __call__
> response = self.handle_request(request)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/perfmetrics-2.0-py2.7.egg/perfmetrics/__init__.py',
> line 270 in handle
> return handler(request)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/slowlog-0.9-py2.7.egg/slowlog/tween.py',
> line 38 in __call__
> return self.handler(request)
> File
> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/tweens.py',
> line 17 in excview_tween
> response = handler(request)
> File
> '/Users/pa...

[Paul Everitt (paul-agendaless)]

Sorry, I wasn't clear. This was in my local buildout. I did bin/karlserve evolve.

--Paul

On Aug 22, 2014, at 3:33 PM, Chris Rossi <email address hidden> wrote:

> There's an evolve step for this one. Did you use the osideploy script?
>
> Chris
>
>
> On Fri, Aug 22, 2014 at 2:34 PM, Paul Everitt <email address hidden> wrote:
>
>> This also had a couple of similar tracebacks, include _community_cloud.
>> Both pasted below, occurred when I tagged something.
>>
>> URL: http://localhost:6543/pg/tagcloud.html
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/Paste-1.7.5.1-py2.7.egg/paste/evalexception/middleware.py',
>> line 306 in respond
>> app_iter = self.application(environ, detect_start_response)
>> File '/Users/paul/projects/karl/dev-buildout/src/karl/karl/timeit.py',
>> line 48 in __call__
>> resp = req.get_response(self.app)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py',
>> line 1086 in get_response
>> application, catch_exc_info=False)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py',
>> line 1055 in call_application
>> app_iter = application(self.environ, start_response)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/repoze.browserid-0.3-py2.7.egg/repoze/browserid/middleware.py',
>> line 127 in __call__
>> return self.app(environ, start_response)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py',
>> line 176 in __call__
>> response = self.handle_request(request)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/tweens.py',
>> line 17 in excview_tween
>> response = handler(request)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py',
>> line 153 in handle_request
>> response = view_callable(context, request)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py',
>> line 319 in viewresult_to_response
>> result = view(context, request)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py',
>> line 402 in _requestonly_view
>> response = view(request)
>> File
>> '/Users/paul/projects/karl/dev-buildout/src/karlserve/karlserve/application.py',
>> line 135 in site_dispatch
>> return request.get_response(instance.pipeline())
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py',
>> line 1086 in get_response
>> application, catch_exc_info=False)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/WebOb-1.1.1-py2.7.egg/webob/request.py',
>> line 1055 in call_application
>> app_iter = application(self.environ, start_response)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/pyramid-1.2.1-py2.7.egg/pyramid/router.py',
>> line 176 in __call__
>> response = self.handle_request(request)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/perfmetrics-2.0-py2.7.egg/perfmetrics/__init__.py',
>> line 270 in handle
>> return handler(request)
>> File
>> '/Users/paul/projects/karl/dev-buildout/eggs/slowlog-0.9-py2.7.egg/slowlog/tween.py',...