ValueError in pgtextindex getContextualSummaries

[Paul Everitt]

This URL: https://karl.soros.org/searchresults.html?body=formstack&kind=files

...generates this error:

ValueError: A string literal cannot contain NUL (0x00) characters.

FTR, there are quite a few exceptions in the error log from the last few days. Looks like 4 different kinds of errors.

Here's the traceback:

Traceback (most recent call last):
  File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid-1.2.1-py2.7.egg/pyramid/tweens.py", line 17, in excview_tween
    response = handler(request)
  File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py", line 107, in tm_tween
    return response
  File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py", line 75, in __exit__
    return self._retry_or_raise(t, v, tb)
  File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py", line 60, in _retry_or_raise
    reraise(t, v, tb) # otherwise reraise the exception
  File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py", line 100, in tm_tween
    response = handler(request)
  File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid-1.2.1-py2.7.egg/pyramid/router.py", line 153, in handle_request
    response = view_callable(context, request)
  File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py", line 186, in _secured_view
    return view(context, request)
  File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py", line 292, in rendered_view
    result = view(context, request)
  File "/srv/osfkarl/production/113/karl/views/search.py", line 174, in searchresults_view
    show_search_knobs=True,
  File "/srv/osfkarl/production/113/karl/views/search.py", line 351, in _searchresults_view
    description = summarizer(doc, text_term)
  File "/srv/osfkarl/production/113/karl/models/adapters.py", line 612, in __call__
    doc_text, query, MaxFragments=3)
  File "/srv/osfkarl/.buildout/eggs/cp27mu/repoze.pgtextindex-1.4-py2.7.egg/repoze/pgtextindex/index.py", line 376, in get_contextual_summary
    return self.get_contextual_summaries([raw_text], query, **options)[0]
  File "/srv/osfkarl/.buildout/eggs/cp27mu/repoze.pgtextindex-1.4-py2.7.egg/repoze/pgtextindex/index.py", line 401, in get_contextual_summaries
    cursor.execute(stmt, params + tuple(raw_texts))
ValueError: A string literal cannot contain NUL (0x00) characters.

[Carlos de la Guardia (cguardia)]

The error is easily replicated. Taking a look.

[Carlos de la Guardia (cguardia)]

This was a bit of a rabbit hole, since when I tried to test this I found that staging search was not working. Once I fixed that (see separate email), I found that this file was causing the problem:

https://karl.soros.org/communities/resource-center/files/web-files/ifc-pilot/formstack-ifc-embed-code.txt/

I fixed it by copying and pasting the text into a new file, so the encoding was somehow very wrong in the original file.

I could add more code to try to prevent this sort of error, but given that this hadn't occurred previously and is only a problem for search (you could visit the file and see its contents even with the wrong encoding), we could just cross our fingers and hope it doesn't happen again.

What do you think?

[Paul Everitt (paul-agendaless)]

I agree completely, we shouldn’t over-invest in something that hasn’t happened much. Thanks! I guess mark this as Fix Released.

—Paul

> On Feb 15, 2018, at 4:08 AM, Carlos de la Guardia <email address hidden> wrote:
>
> This was a bit of a rabbit hole, since when I tried to test this I found
> that staging search was not working. Once I fixed that (see separate
> email), I found that this file was causing the problem:
>
> https://karl.soros.org/communities/resource-center/files/web-files/ifc-
> pilot/formstack-ifc-embed-code.txt/
>
> I fixed it by copying and pasting the text into a new file, so the
> encoding was somehow very wrong in the original file.
>
> I could add more code to try to prevent this sort of error, but given
> that this hadn't occurred previously and is only a problem for search
> (you could visit the file and see its contents even with the wrong
> encoding), we could just cross our fingers and hope it doesn't happen
> again.
>
> What do you think?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1749539
>
> Title:
> ValueError in pgtextindex getContextualSummaries
>
> Status in KARL4:
> New
>
> Bug description:
> This URL:
> https://karl.soros.org/searchresults.html?body=formstack&kind=files
>
> ...generates this error:
>
> ValueError: A string literal cannot contain NUL (0x00) characters.
>
> FTR, there are quite a few exceptions in the error log from the last
> few days. Looks like 4 different kinds of errors.
>
> Here's the traceback:
>
> Traceback (most recent call last):
> File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid-1.2.1-py2.7.egg/pyramid/tweens.py", line 17, in excview_tween
> response = handler(request)
> File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py", line 107, in tm_tween
> return response
> File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py", line 75, in __exit__
> return self._retry_or_raise(t, v, tb)
> File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py", line 60, in _retry_or_raise
> reraise(t, v, tb) # otherwise reraise the exception
> File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid_tm-0.5-py2.7.egg/pyramid_tm/__init__.py", line 100, in tm_tween
> response = handler(request)
> File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid-1.2.1-py2.7.egg/pyramid/router.py", line 153, in handle_request
> response = view_callable(context, request)
> File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py", line 186, in _secured_view
> return view(context, request)
> File "/srv/osfkarl/.buildout/eggs/cp27mu/pyramid-1.2.1-py2.7.egg/pyramid/config/views.py", line 292, in rendered_view
> result = view(context, request)
> File "/srv/osfkarl/production/113/karl/views/search.py", line 174, in searchresults_view
> show_search_knobs=True,
> File "/srv/osfkarl/production/113/karl/views/search.py", line 351, in _searchresults_view
> description = summarizer(doc, text_term)
> File "/srv/osfkarl/produ...