(OXFAM) tagusers error, error in search path

[Jim B. Glenn]

reported via error monitor:

ENTRY
Fri Oct 22 08:32:58 2010
Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=shop&docid=-589375072
Traceback (most recent call last):
  File "/opt/karl/oxfam/3.5.4-0/src/karl/karl/errorlog.py", line 15, in middleware
    return app(environ, start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.zodbconn-0.11-py2.5.egg/repoze/zodbconn/connector.py", line 21, in __call__
    result = self.next_app(environ, start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.retry-0.9.4-py2.5.egg/repoze/retry/__init__.py", line 88, in __call__
    app_iter = self.application(environ, replace_start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.tm2-1.0a5-py2.5.egg/repoze/tm/__init__.py", line 23, in __call__
    result = self.application(environ, save_status_and_headers)
  File "/opt/karl/oxfam/3.5.4-0/eggs/Paste-1.7.2-py2.5.egg/paste/registry.py", line 350, in __call__
    app_iter = self.application(environ, start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.who-1.0.15-py2.5.egg/repoze/who/middleware.py", line 107, in __call__
    app_iter = app(environ, wrapper.wrap_start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.browserid-0.3-py2.5.egg/repoze/browserid/middleware.py", line 127, in __call__
    return self.app(environ, start_response)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/router.py", line 130, in __call__
    response = view_callable(context, request)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/configuration.py", line 1753, in _secured_view
    return view(context, request)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/configuration.py", line 1704, in _rendered_view
    response = view(context, request)
  File "/opt/karl/oxfam/3.5.4-0/src/karl/karl/views/tags.py", line 402, in tag_users_view
    target = find_model(context, address)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/traversal.py", line 75, in find_model
    D = traverse(model, path)
  File "/opt/karl/oxfam/3.5.4-0/eggs/repoze.bfg-1.2-py2.5.egg/repoze/bfg/traversal.py", line 276, in traverse
    request = Request.blank(path)
  File "/opt/karl/oxfam/3.5.4-0/eggs/WebOb-0.9.8.0_osi1-py2.5.egg/webob/request.py", line 943, in blank
    if SCHEME_RE.search(path):
TypeError: expected string or buffer

[Paul Everitt (paul-agendaless)]

This is a pretty odd one. Over to Tres for triage.

[Jim B. Glenn (jimbglenn)]

reported again multiple times via error monitor on

Tue Oct 26 06:38:46 2010 Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=march5&docid=658437439

Tue Oct 26 06:43:38 2010 Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=book&docid=-806746457

Tue Oct 26 06:47:01 2010 Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=team&docid=658437439

Tue Oct 26 06:50:29 2010 Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=debate&docid=-806746455

Tue Oct 26 06:51:39 2010 Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=ukc&docid=658437439

Tue Oct 26 06:51:50 2010 Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=dicussion&docid=-806746457

Tue Oct 26 06:51:54 2010 Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=elephant&docid=-806746457

Tue Oct 26 06:51:55 2010 Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=elephant&docid=-806746455

Tue Oct 26 06:54:20 2010 Exception when processing https://karl.oxfam.org.uk/tagusers.html?tag=lakoff&docid=-806746457

.....

[Tres Seaver (tseaver)]

Hmm, this pretty much means that we have docids in the catalog whose
paths no longer correspond to real content. Jim, can you please run the
following to find such content, and paste the output here (or append as an
attachment, if it is too long)?

  $ bin/debug
  >>> from repoze.bfg.traversal import find_model
  >>> map = root.catalog.document_map
  >>> for path in map.address_to_docid.keys():
  ... try:
  ... found = find_model(root, path)
  ... except KeyError:
  ... print path

[Tres Seaver (tseaver)]

Crap, the formatting seems hosed on that example::

  $ bin/debug
  >>> from repoze.bfg.traversal import find_model
  >>> map = root.catalog.document_map
  >>> for path in map.address_to_docid.keys():
  ... try:
  ... found = find_model(root, path)
  ... except KeyError:
  ... print path

[Tres Seaver (tseaver)]

Ugh, the 'try:' and 'except KeyError:' lines should be indented four spaces,
and the 'found =' and 'print path' lines eight spaces.

[Jim B. Glenn (jimbglenn)]

Took a while to run, but didn't print any result:

[oxfam@karlhost01 current]$ bin/debug
Python 2.5.5 (r255:77872, Jun 7 2010, 14:10:57)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> from repoze.bfg.traversal import find_model
>>> map = root.catalog.document_map
>>> for path in map.address_to_docid.keys():
... try:
... found = find_model(root, path)
... except KeyError:
... print path
...
>>>
>>>

[Jim B. Glenn (jimbglenn)]

ran on STAGING

[Tres Seaver (tseaver)]

OK, please try this instead::

  >>> from repoze.bfg.traversal import find_model
  >>> map = root.catalog.document_map
  >>> for docid, path in map.docid_to_address.items():
  ... try:
  ... found = find_model(root, path)
  ... except KeyError:
  ... print docid, path

[Jim B. Glenn (jimbglenn)]

ran on staging:

no results:

[oxfam@karlhost01 current]$ bin/debug
Python 2.5.5 (r255:77872, Jun 7 2010, 14:10:57)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help" for more information. "root" is the karl root object.
>>> from repoze.bfg.traversal import find_model
>>> map = root.catalog.document_map
>>> for docid, path in map.docid_to_address.items():
... try:
... found = find_model(root, path)
... except KeyError:
... print docid, path

...
>>> >>>
>>>
>>>

[Tres Seaver (tseaver)]

OK, if we can't find junk data in the catalog to explain passing
a non-string down into 'find_model', and we can't reproduce this
bug, then we can't do anything to fix it.

Feel free to reopen if the problem recurs, or if more information
becomes available.