Report page crashes with UTF8 in function

[Ingeborg Hellemo]

Nav 4.2.3

While making reports via /report, if you click on an item in the function column which contains special characters (æøå) you get

DataError at /report/netbox

invalid byte sequence for encoding "UTF8": 0xf8

Request Method: GET
Request URL: https://nav.uit.no/report/netbox?function=Tilh%C3%B8rer%20PET-scanner%20for%20dyrefors%C3%B8k
Django Version: 1.4.20
Exception Type: DataError
Exception Value:

invalid byte sequence for encoding "UTF8": 0xf8

Exception Location: /usr/local/lib/python2.7/nav/report/dbresult.py in __init__, line 49
Python Executable:
Python Version: 2.7.9
Python Path:

['/usr/local/lib/python2.7/site-packages/django_oauth2_provider-0.2.6-py2.7.egg',
 '/usr/local/lib/python2.7/site-packages/shortuuid-0.4.2-py2.7.egg',
 '/usr/local/lib/python2.7/site-packages/djangorestframework-2.3.8-py2.7.egg',
 '/usr/local/lib/python2.7/site-packages/django_filter-0.7-py2.7.egg',
 '/usr/local/lib/python2.7/site-packages/django_hstore-1.2.5-py2.7.egg',
 '/usr/local/lib/python2.7/site-packages/crispy_forms_foundation-0.2.3-py2.7.egg',
 '/usr/local/lib/python2.7/site-packages/django_crispy_forms-1.3.2-py2.7.egg',
 '/usr/local/lib/python2.7/site-packages/Twisted-12.3.0-py2.7-freebsd-10.0-RELEASE-p9-amd64.egg',
 '/usr/local/lib/python27.zip',
 '/usr/local/lib/python2.7',
 '/usr/local/lib/python2.7/plat-freebsd10',
 '/usr/local/lib/python2.7/lib-tk',
 '/usr/local/lib/python2.7/lib-old',
 '/usr/local/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/site-packages',
 '/usr/local/lib/python2.7/site-packages/PIL',
 '/usr/local/nav/etc/python',
 '/usr/local/lib/python2.7/nav/statemon/checker']

[Morten Brekkevold (mbrekkevold)]

I am unable to reproduce this, Ingeborg. I have psycopg2 2.4.5 and PostgreSQL 9.1.14.

Only thing that might help is setting DJANGO_DEBUG=True in `nav.conf`, provoke the error and dump a proper traceback from the web page, with the local vars of at least /usr/local/lib/python2.7/nav/report/dbresult.py line 49.

[Ingeborg Hellemo (ingeborg-hellemo)]

Sloppy reporting on my side. Sorry! Do you need even more (META? SETTINGS?)

On NAV-server
py27-psycopg2-2.5.5 High performance Python adapter for PostgreSQL
postgresql93-client-9.3.6_1 PostgreSQL database (client)

On DB-server:
postgresql92-server-9.2.10 The most advanced open-source database
available anywhere

(Can the difference i version between client and server be an issue?)

Tracebak (copy-and-paste view)

Environment:

Request Method: GET
Request URL: https://nav.uit.no/report/netbox?function=Tilh%C3%B8rer%20PET-scan
ner%20for%20dyrefors%C3%B8k

Django Version: 1.4.20
Python Version: 2.7.9
Installed Applications:
('nav.models',
 'nav.django',
 'django.contrib.staticfiles',
 'django.contrib.sessions',
 'django.contrib.humanize',
 'crispy_forms',
 'crispy_forms_foundation',
 'django_hstore',
 'rest_framework')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'nav.django.auth.AuthenticationMiddleware',
 'nav.django.auth.AuthorizationMiddleware',
 'nav.django.legacy.LegacyCleanupMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware')

Traceback:
File "/usr/local/lib/python2.7/site-packages/django/core/handlers/base.py" in
get_response
  109. response = callback(request, *callback_args,
**callback_kwargs)
File "/usr/local/lib/python2.7/nav/web/report/views.py" in get_report
  77. return make_report(request, report_name, export_delimiter, query)
File "/usr/local/lib/python2.7/nav/web/report/views.py" in make_report
  287. result_time) = _fetch_data_from_db()
File "/usr/local/lib/python2.7/nav/web/report/views.py" in _fetch_data_from_db
  258. CONFIG_FILE_LOCAL, query_dict, None, None))
File "/usr/local/lib/python2.7/nav/report/generator.py" in make_report
  83. dbresult = DatabaseResult(config)
File "/usr/local/lib/python2.7/nav/report/dbresult.py" in __init__
  49. database.execute(self.sql)

Exception Type: DataError at /report/netbox
Exception Value: invalid byte sequence for encoding "UTF8": 0xf8

[Morten Brekkevold (mbrekkevold)]

I think this is related to bug 1233093.

NAV specifies psycopg2==2.4.5 as a dependency, but you are running 2.5.1. When testing with psycopg2==2.5.1 (same as yours), I also get a crash.

 You must have some local patch installed for this, though, because I get a different crash from you:

Traceback:
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py" in get_response
  109. response = callback(request, *callback_args, **callback_kwargs)
File "/source/python/nav/web/report/views.py" in get_report
  77. return make_report(request, report_name, export_delimiter, query)
File "/source/python/nav/web/report/views.py" in make_report
  287. result_time) = _fetch_data_from_db()
File "/source/python/nav/web/report/views.py" in _fetch_data_from_db
  258. CONFIG_FILE_LOCAL, query_dict, None, None))
File "/source/python/nav/report/generator.py" in make_report
  73. (contents, neg, operator) = arg_parser.parse_query(args)
File "/source/python/nav/report/generator.py" in parse_query
  318. value = intstr(value)
File "/source/python/nav/report/generator.py" in intstr
  382. return nav.db.escape(arg)
File "/source/python/nav/db/__init__.py" in escape
  84. return str(psycopg2.extensions.QuotedString(string, "UTF8"))

Exception Type: TypeError at /report/netbox
Exception Value: function takes exactly 1 argument (2 given)

[Ole Martin Bjørndalen (olemb)]

Yes, this was caused by my patch for bug 1443812. I've removed the patch and downgraded to 2.4.5 and this fixed both problems. I'll update the other bug report as well.

[Ole Martin Bjørndalen (olemb)]

My patch for bug 1233093, I mean.