Live Search Not Producing Expected Result

[Anthony]

Bug Reported by Jonathan Hooper:

When I type the search term "fgrep" I don’t see any results, but a blog comment clearly contains this phrase:

https://karl.soros.org/communities/hoops-snips/blog/recursive-grep

Do we know why this is happening?

[Paul Everitt (paul-agendaless)]

Don't know if this is better for you or Shane. If you want, re-assign it.

[Chris McDonough (chrism-plope)]

This object converts its text to html when it's indexed, and also makes use of the title in the text index:

>>> ob = root['communities']['hoops-snips']['blog']['recursive-grep']['comments']['002']
>>> ob.text
u'<div xmlns="http://www.w3.org/1999/xhtml">\r\n <p>recursive grep, then delete<br><br>fgrep -lir baltimore * | xargs rm</p>\r\n </div>\n'
>>> from karl.utilities.converters.stripogram import html2text
>>> html2text(ob.text)
u'\n\nrecursive grep, then delete\n\n\n\nfgrep -lir baltimore * | xargs rm'
>>> ob.title
u'Recursive GREP'

For this object we've indexed the following words:

>>> docid = root['communities']['hoops-snips']['blog']['recursive-grep']['comments']['002'].docid
>>> wids = root.catalog['texts'].index.get_words(docid)
>>> index = texts = root.catalog['texts'].index
>>> lexicon = index._lexicon
>>> map(lexicon.get_word, wids)
[u'recursive', u'grep', u'recursive', u'grep', u'delete']

I can't make much sense out of that yet.

[Chris McDonough (chrism-plope)]

Found it. The stripogram stuff is a decoy. The actual implementation uses karl.content.models.adapters._html_cleaner, which has a bug:

>>> from karl.content.models.adapters import _html_cleaner
>>> _html_cleaner(u'<div xmlns="http://www.w3.org/1999/xhtml">\r\n <p>recursive grep, then delete<br><br>fgrep -lir baltimore * | xargs rm</p>\r\n </div>\n')

Returns:

'\r\n recursive grep, then delete'

It should return something more like the previous stripogram example (u'\n\nrecursive grep, then delete\n\n\n\nfgrep -lir baltimore * | xargs rm').

[Chris McDonough (chrism-plope)]

The bug which caused improper input to the text indexer has been fixed on the trunk.

We'll need to run "bin/reindex_catalog" on the production system after the next release to fix existing content.

[Paul Everitt (paul-agendaless)]

Hi Chris. Looks like this didn't get done completely in production. Some old content with fgrep isn't searchable.

[Chris Rossi (chris-archimedeanco)]

Hi Paul, Chris M's fix is from 9/22 and the last release of Karl was 9/8, so we're still waiting for this to go into production.