ascii decode problem on execute

[spencer rathbun]

When inserting data into a test Mysql db, I ran into the following issue:

  File "C:\development\taxStatements\tax converter\taxes.py", line 60, in main
    db.insertRec(rec)
  File "C:\development\taxStatements\tax converter\dbInserter.py", line 1735, in insertRec
    self.insertManRec(rec, firstrec)
  File "C:\development\taxStatements\tax converter\dbInserter.py", line 1772, in insertManRec
    rec['Owner_City'], rec['Owner_State'], rec['Owner_Zip_Code'], rec['Owner_Country'], rec['Ow
ner_Attention_Line'] \
  File "C:\development\taxStatements\tax converter\dbInserter.py", line 1718, in get_or_create_
man
    instance = self.session.query(model).get((name, st, st_num, st_dir, st_suf, city, state, zi
p_code, country, attn_line))
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 775, in get
    return self._load_on_ident(key)
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2512, in _load_on_ident
    return q.one()
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2184, in one
    ret = list(self)
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2227, in __iter__
    return self._execute_and_instances(context)
  File "C:\Python27\lib\site-packages\sqlalchemy\orm\query.py", line 2242, in _execute_and_inst
ances
    result = conn.execute(querycontext.statement, self._params)
  File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 1449, in execute
    params)
  File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 1584, in _execute_clause
element
    compiled_sql, distilled_params
  File "C:\Python27\lib\site-packages\sqlalchemy\engine\base.py", line 1691, in _execute_contex
t
    context)
  File "C:\Python27\lib\site-packages\sqlalchemy\dialects\mysql\oursql.py", line 96, in do_exec
ute
    cursor.execute(statement, parameters)
  File "cursor.pyx", line 122, in oursql.Cursor.execute (oursqlx\oursql.c:15747)
  File "statement.pyx", line 318, in oursql._Statement.execute (oursqlx\oursql.c:9241)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 13: ordinal not in range(1
28)

I'm using Python 2.7.2, Sqlalchemy 7.8, and oursql 9.3 on a Win 7 64bit system. I'm reading in a data file in a simple for loop, doing no encoding to the data. I convert each line into a dictionary, then create a new object with the sqlalchemy orm. It converts that object into an insert statement for the oursql library, which fails with this encoding issue when I have non-ascii elements in my data.

Line 318: http://bazaar.launchpad.net/~jerub/oursql/master/view/head:/oursqlx/statement.pyx#L318

Looking at the source, it seems that the cursor just assumes that it will receive unicode objects, not strings. Since this check is inside of the use_unicode if statement, disabling that option avoids the problem. However, data is lost in this case.

This answer on SO, http://stackoverflow.com/a/11544648/724357, seems to indicate that decoding the input strings with the proper encoding will fix the issue. Decoding does solve this data issue.

Can the documentation be updated to reflect that use_unicode=0 may cause data loss, and that use_unicode=1 expects a unicode object be passed in, not a string?