DatabaseError: invalid byte sequence for encoding "UTF8": 0x80

[Lars Butler]

This is a very similar error to bug # 911714.

test_serialize (tests.input.source_test.SourceDBWriterTestCase) ... ERROR

======================================================================
ERROR: test_serialize (tests.input.source_test.SourceDBWriterTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/larsbutler/proj/oq-engine/tests/input/source_test.py", line 263, in test_serialize
    db_writer.serialize()
  File "/Users/larsbutler/.virtualenvs/openquake/lib/python2.7/site-packages/django/db/transaction.py", line 217, in inner
    res = func(*args, **kwargs)
  File "/Users/larsbutler/proj/oq-engine/openquake/input/source.py", line 357, in serialize
    ps.save()
  File "/Users/larsbutler/.virtualenvs/openquake/lib/python2.7/site-packages/django/db/models/base.py", line 460, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/Users/larsbutler/.virtualenvs/openquake/lib/python2.7/site-packages/django/db/models/base.py", line 553, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/Users/larsbutler/.virtualenvs/openquake/lib/python2.7/site-packages/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/Users/larsbutler/.virtualenvs/openquake/lib/python2.7/site-packages/django/db/models/query.py", line 1436, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/larsbutler/.virtualenvs/openquake/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 791, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "/Users/larsbutler/.virtualenvs/openquake/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 735, in execute_sql
    cursor.execute(sql, params)
  File "/Users/larsbutler/.virtualenvs/openquake/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)
DatabaseError: invalid byte sequence for encoding "UTF8": 0x80

I encountered this error while running tests on this branch: https://github.com/larsbutler/oq-engine/tree/2995b00f3a77cf9c71119a06e3028db55a22ab23

[Lars Butler (lars-butler)]

Discovered on OSX, with Django 1.3 and Postgres 9.1.1.

Bug was confirmed by andreacerisara on OSX with Django 1.3.1 and Postgres 9.1.

I need to test this on Ubuntu to try to narrow down the source of the issue.

[Muharem Hrnjadovic (al-maisan)]

hzrdi.parsed_source.blob needs to be of type BYTEA, for the django model changes please see: http://stackoverflow.com/questions/517349/binaryfields-in-django-models

[Lars Butler (lars-butler)]

This can actually be fixed without changing the db field type. It works with this patch to openquake/db/models.py:
http://pastie.org/4066428

I took a hint from this thread: http://stackoverflow.com/questions/4915397/django-blob-model-field