ascii error if an accent in data

[Dewinne Julien]

Client Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/web/common/http.py", line 180, in dispatch
    response["result"] = method(controller, self, **self.params)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/web/controllers/main.py", line 1004, in call_button
    action = self.call_common(req, model, method, args, domain_id, context_id)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/web/controllers/main.py", line 948, in call_common
    return self._call_kw(req, model, method, args, {})
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/web/controllers/main.py", line 962, in _call_kw
    return getattr(req.session.model(model), method)(*args, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/web/common/openerplib/main.py", line 250, in proxy
    args, kw)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/web/common/openerplib/main.py", line 117, in proxy
    result = self.connector.send(self.service_name, method, *args)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/web/common/http.py", line 611, in send
    raise fault

Server Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/web/common/http.py", line 592, in send
    result = openerp.netsvc.dispatch_rpc(service_name, method, args)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/netsvc.py", line 360, in dispatch_rpc
    result = ExportService.getService(service_name).dispatch(method, params)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/service/web_services.py", line 572, in dispatch
    res = fn(db, uid, *params)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/osv/osv.py", line 167, in execute_kw
    return self.execute(db, uid, obj, method, *args, **kw or {})
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/osv/osv.py", line 121, in wrapper
    return f(self, dbname, *args, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/osv/osv.py", line 176, in execute
    res = self.execute_cr(cr, uid, obj, method, *args, **kw)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/osv/osv.py", line 164, in execute_cr
    return getattr(object, method)(cr, uid, *args, **kw)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/ea_import/ea_import_chain.py", line 61, in import_to_db
    csv_list = chain.file_to_list()
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/osv/orm.py", line 368, in function_proxy
    return attr(self._cr, self._uid, [self._id], *args, **kwargs)
  File "/usr/local/lib/python2.6/dist-packages/openerp-6.1_1-py2.6.egg/openerp/addons/ea_import/ea_import_chain.py", line 89, in file_to_list
    for field_string in string_record.split(chain.separator):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 26: ordinal not in range(128)

[Dewinne Julien (julien-dewinne)]

Hi,

This message appears when we have an accent in the data than we want to import in OpenERP.

Regards,

BHC team

[Enapps Ltd OpenERP Partner London UK (enapps-london-uk)]

ok thank you for reporting, you may need a double-quote separator ... not tried this yet as we don't have accent characters to deal with. However the decoding error suggests that there is indeed a bug to fix.

[Dewinne Julien (julien-dewinne)]

Hi,

We have found a correction to this bug.

If we modify the file "ea_import_chain.py" like this:

line 82:

    def file_to_list(self, cr, uid, ids, context={}):
        result = []
        cpt=0
        for chain in self.browse(cr, uid, ids, context={}):
            input_file = StringIO(base64.b64decode(chain.input_file))
            for string_record in input_file:
                line = []
 + string_record=string_record.decode('latin-1')
                for field_string in string_record.split(chain.separator):
                    field_string.encode('latin-1')
                    delimiter = chain.delimiter and chain.delimiter + "\n\r" or "\n\r"
                    line.append(field_string.strip(delimiter))
                result.append(line)
        return result

With that, the accents is correctly imported in the DB.

[Dewinne Julien (julien-dewinne)]

I forgot a + in my last comment on this line:

+ field_string.encode('latin-1')

[Enapps Ltd OpenERP Partner London UK (enapps-london-uk)]

great thank you for the feedback. I am just in the middle of a few things but I will test the latin-1 encoding and if all OK, implement the fix ASAP.

many thanks
Colin

[jmddd (jeanmarie-duminger)]

I had to use 'utf-8' instead of 'latin-1' to get a correct rendering of the imported characters with accents. But the suggested solution seems efficient. Thanks.
JM

[Juan Rial (OpenERP) (jri-openerp)]

Fixed it in lp:~jri-openerp/+junk/enapps-csv-import-tool-trunk by adding a charset selection field. Currently it's populated with a couple of usual suspects; adding more obscure formats is pretty straightforward.

The patch also makes the separator and delimiter fields mandatory as I believe they should be.

For some reason I don't see the "propose merge" button anywhere in my branch page, so patch in attachment.

[Enapps Ltd OpenERP Partner London UK (enapps-london-uk)]

thanks for helping all. we have a few more fixes to upload soon.

[Enapps Ltd OpenERP Partner London UK (enapps-london-uk)]

fixed in latest commit