UnicodeDecodeError in UnicodeReflectionTest

[Mathieu Fenniak]

In SQLAlchemy's engine.reflection.UnicodeReflectionTest.test_basic, a UnicodeDecodeError occurs:

    'ascii' codec can't decode byte 0xe6 in position 0: ordinal not in range(128)

This could be related to pg8000 treating Pg's SQL_ASCII encoding as the same as "ascii", while the documentation states that they are slightly different. sql_ascii should treat bytes 128-256 as unknown bytes.

[geophile (jao-geophile)]

I ran into something very similar. Here is the query:

select 'mm1_4_0', region_usage from mm1_4_0.region_timestamp union all
select 'mm1_4_5', region_usage from mm1_4_5.region_timestamp union all
select 'mm1_4_8', region_usage from mm1_4_8.region_timestamp union all
select 'mm1_4_a', region_usage from mm1_4_a.region_timestamp union all
select 'mm1_4_d', region_usage from mm1_4_d.region_timestamp union all
select 'mm1_4_f', region_usage from mm1_4_f.region_timestamp

The region_timestamp.region_usage column is declared to be of type char, and it always stores one of 'A', 'B', 'I'.

Here is the pg8000 part of the stack:

  File "/home/jao/osh-1.0.0/osh/external/pg8000/dbapi.py", line 304, in execute
    self._execute(operation, args)
  File "/home/jao/osh-1.0.0/osh/external/pg8000/dbapi.py", line 309, in _execute
    self.cursor.execute(new_query, *new_args)
  File "/home/jao/osh-1.0.0/osh/external/pg8000/interface.py", line 297, in execute
    self._stmt = PreparedStatement(self.connection, query, statement_name="", *[type(x) for x in args])
  File "/home/jao/osh-1.0.0/osh/external/pg8000/interface.py", line 98, in __init__
    self._parse_row_desc = self.c.parse(self._statement_name, statement, types)
  File "/home/jao/osh-1.0.0/osh/external/pg8000/protocol.py", line 857, in _fn
    return fn(self, *args, **kwargs)
  File "/home/jao/osh-1.0.0/osh/external/pg8000/protocol.py", line 963, in parse
    self._send(Parse(statement, qs, param_types))
  File "/home/jao/osh-1.0.0/osh/external/pg8000/protocol.py", line 905, in _send
    data = msg.serialize()
  File "/home/jao/osh-1.0.0/osh/external/pg8000/protocol.py", line 120, in serialize
    val = struct.pack("!i", len(val) + 4) + val
UnicodeDecodeError: 'ascii' codec can't decode byte 0xac in position 3: ordinal not in range(128)

[Mariano Reingart (reingart)]

I've made a maintenance fork:

http://code.google.com/p/pg8000/

Could you test the latest version?

Several unicode issues were fixed.
If the error persist, please could you report it in the googlecode project issues?
(some data to build a unittest would be great)