Error in accented characters encoding in received mail

[Sharoon Thomas http://openlabs.co.in]

      * Display problem with accented characters, here is an example of mail downloaded:
           Je l'ai d=E9ja envoy=E9 un e-mail et j'attends encore sa r=E9ponse.
           D=E9sol=E9 pour le d=E9lai mis pour r=E9pondre =E0 vos mails, je suis trop= pris par la charge de travail que j'ai
        (=E9 is é ; =E0 is à)

[Sharoon Thomas http://openlabs.co.in (sharoonthomas)]

Can you forward a sample email to me..

As a matter of fact i never get emails in a language with accented characters to even test

[Sharoon Thomas http://openlabs.co.in (sharoonthomas)]

Closing the issue, since there has been no update

[Carlo - Didotech.com (iw3hxn)]

I have the same problem:

email text in italian:
Il messaggio è pronto per l'invio con i seguenti file o collegamenti allegati:it_IT.po
Nota: per proteggere il computer dai virus, le applicazioni di posta elettronica impediscono l'invio o la ricezione di alcuni tipi di allegati. Per determinare la modalità di gestione degli allegati, controllare le impostazioni di protezione della posta elettronica.

and the error in mail download is:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/openerp-server/netsvc.py", line 245, in dispatch
    result = LocalService(service_name)(method, *params)
  File "/usr/lib/python2.6/site-packages/openerp-server/netsvc.py", line 74, in __call__
    return getattr(self, method)(*params)
  File "/usr/lib/python2.6/site-packages/openerp-server/service/web_services.py", line 576, in execute
    res = service.execute(db, uid, object, method, *args)
  File "/usr/lib/python2.6/site-packages/openerp-server/osv/osv.py", line 58, in wrapper
    return f(self, dbname, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/openerp-server/osv/osv.py", line 119, in execute
    res = pool.execute_cr(cr, uid, obj, method, *args, **kw)
  File "/usr/lib/python2.6/site-packages/openerp-server/osv/osv.py", line 111, in execute_cr
    return getattr(object, method)(cr, uid, *args, **kw)
  File "/home/openerp/addons/poweremail/poweremail_mailbox.py", line 152, in complete_mail
    self.pool.get('poweremail.core_accounts').get_fullmail(cr, uid, id, context)
  File "/home/openerp/addons/poweremail/poweremail_core.py", line 1019, in get_fullmail
    server_ref, mailid, context)
  File "/home/openerp/addons/poweremail/poweremail_core.py", line 730, in complete_mail
    parsed_mail = self.get_payloads(mail)
  File "/home/openerp/addons/poweremail/poweremail_core.py", line 1050, in get_payloads
    parsed_mail['text'] = tools.ustr(part.get_payload(decode=True)) # decode=True to decode a MIME message
  File "/usr/lib/python2.6/site-packages/openerp-server/tools/misc.py", line 746, in ustr
    raise UnicodeError('unable de to convert %r' % (orig,))
UnicodeError: unable de to convert " <<it_IT.po>> \nIl messaggio \xe8 pronto per l'invio con i seguenti file o collegamenti allegati:\n\nit_IT.po\n\n\nNota: per proteggere il computer dai virus, le applicazioni di posta elettronica impediscono l'invio o la ricezione di alcuni tipi di allegati. Per determinare la modalit\xe0 di gestione degli allegati, controllare le impostazioni di protezione della posta elettronica.\n"

[Sharoon Thomas http://openlabs.co.in (sharoonthomas)]

I am not really sure if the error is with the email decoder or with ustr.

ustr actually raises the exception? maybe its a case where something is not handled.

BTW, i dint really like the idea of raising an exception if conversion fails.

[End of the Day: The ultimate solution resides in Python 3 native unicode handling]

[Carlo - Didotech.com (iw3hxn)]

I don't know what you mean exacly with your reply. anyway emails text are not always encoded in utf8. also other charset like for example latin-1 are used.

[Richard (John) Nopkins@Servosoft (richard-nopkins)]

Its not a bug related to openobject-server.
May I close it?

[Sharoon Thomas http://openlabs.co.in (sharoonthomas)]

Hello John,

According to me this is an issue with openobject-server because ustr should never raise an exception. This change was introduced in 5.0.8 if I am not wrong.

[Richard (John) Nopkins@Servosoft (richard-nopkins)]

Hi Sharoon,

If it is so, invalidate it with the other project. attach a link here where to get this module. i would try to see more.
thanks for your response.

[Carlo - Didotech.com (iw3hxn)]

Hi John,

it's the link of module: http://openlabs.co.in/tmp/poweremail.zip

if you need something i'm here ;)

thanks

[Dominique Chabord (dominique-chabord)]

Hi,
so what is the bottom line on this question ?
I just added power-email to http://dev5.sisalp.net server.
I configured a gmail account with IMAP protocol - wonderful
I cannot download a message with an accent. Same traceback as above.
I may have missed something else too.
Thank you for your help.
regards

[Dominique Chabord (dominique-chabord)]

I tried to encode the message in utf8 and got the same traceback. the traceback is :
Traceback (most recent call last):
  File "/home/sisalpuser/openerp/dev5/server/bin/netsvc.py", line 247, in dispatch
    result = LocalService(service_name)(method, *params)
  File "/home/sisalpuser/openerp/dev5/server/bin/netsvc.py", line 76, in __call__
    return getattr(self, method)(*params)
  File "/home/sisalpuser/openerp/dev5/server/bin/service/web_services.py", line 577, in execute
    res = service.execute(db, uid, object, method, *args)
  File "/home/sisalpuser/openerp/dev5/server/bin/osv/osv.py", line 58, in wrapper
    return f(self, dbname, *args, **kwargs)
  File "/home/sisalpuser/openerp/dev5/server/bin/osv/osv.py", line 119, in execute
    res = pool.execute_cr(cr, uid, obj, method, *args, **kw)
  File "/home/sisalpuser/openerp/dev5/server/bin/osv/osv.py", line 111, in execute_cr
    return getattr(object, method)(cr, uid, *args, **kw)
  File "/home/sisalpuser/openerp/dev5/server/bin/addons/poweremail/poweremail_mailbox.py", line 152, in complete_mail
    self.pool.get('poweremail.core_accounts').get_fullmail(cr, uid, id, context)
  File "/home/sisalpuser/openerp/dev5/server/bin/addons/poweremail/poweremail_core.py", line 973, in get_fullmail
    server_ref, mailid, context)
  File "/home/sisalpuser/openerp/dev5/server/bin/addons/poweremail/poweremail_core.py", line 730, in complete_mail
    parsed_mail = self.get_payloads(mail)
  File "/home/sisalpuser/openerp/dev5/server/bin/addons/poweremail/poweremail_core.py", line 1050, in get_payloads
    parsed_mail['text'] = tools.ustr(part.get_payload(decode=True)) # decode=True to decode a MIME message
  File "/home/sisalpuser/openerp/dev5/server/bin/tools/misc.py", line 746, in ustr
    raise UnicodeError('unable de to convert %r' % (orig,))
UnicodeError: unable de to convert "t\xe9st utf8\r\n\r\n-- \r\nDominique Chabord - SISalp\r\nLogiciel libre pour l'entreprise : Gestion (ERP) et applications web2\r\n18 avenue Beauregard 74960 Cran Gevrier\r\n145A rue Alexandre Borrely 83000 Toulon\r\nt\xe9l [phone] fax [phone] mob [phone]\r\nhttp://sisalp.fr - http://sisalp.org - http://bdll.fr\r\n\r\n"

[Dominique Chabord (dominique-chabord)]

I get the same result on http://stable5.sisalp.net which is the official 5.0.12 version from openerp.
regards

[Nhomar - Vauxoo (nhomar)]

I'm receiving this message.
The probelm appear like is a problem with the management of accents...

Traceback (most recent call last):
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/netsvc.py", line 247, in dispatch
    result = LocalService(service_name)(method, *params)
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/netsvc.py", line 76, in __call__
    return getattr(self, method)(*params)
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/service/web_services.py", line 577, in execute
    res = service.execute(db, uid, object, method, *args)
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/osv/osv.py", line 58, in wrapper
    return f(self, dbname, *args, **kwargs)
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/osv/osv.py", line 119, in execute
    res = pool.execute_cr(cr, uid, obj, method, *args, **kw)
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/osv/osv.py", line 111, in execute_cr
    return getattr(object, method)(cr, uid, *args, **kw)
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/addons/poweremail/poweremail_mailbox.py", line 152, in complete_mail
    self.pool.get('poweremail.core_accounts').get_fullmail(cr, uid, id, context)
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/addons/poweremail/poweremail_core.py", line 1020, in get_fullmail
    server_ref, mailid, context)
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/addons/poweremail/poweremail_core.py", line 731, in complete_mail
    parsed_mail = self.get_payloads(mail)
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/addons/poweremail/poweremail_core.py", line 1051, in get_payloads
    parsed_mail['text'] = tools.ustr(part.get_payload(decode=True)) # decode=True to decode a MIME message
  File "/home/nhomar/openerp/stable/5.0/instancias/netquatro/server/bin/tools/misc.py", line 746, in ustr
    raise UnicodeError('unable de to convert %r' % (orig,))

[Bruno JOLIVEAU - www.savoirfairelinux.com (zeekom)]

Hello,

I met the same problem on 5.0.12

-> Sharoon, i remember you had correct it in the past or i mix up with an other bug ?

Bye,

Bruno

[Richard (John) Nopkins@Servosoft (richard-nopkins)]

Hi sharoon,
thanks.

my colleague reported this few days back.

[Olivier Dony (Odoo) (odo-openerp)]

Closing this on OpenERP server, sorry but tools.ustr() is not the magic answer too all encoding issues, despite what you might think.
You still need to properly know/detect what encoding you are reading strings in, in order to know if tools.ustr() will be able to help you.

BTW the convention is for tools.ustr() to raise an exception when something cannot be converted to unicode, this is much better than silently returning None or u''. In this case it apparently helps highlight an upstream issue, so that's Good!

[dx-vmonroig (dx-vmonroig)]

We're experiencing this problem in v5.0.14, too.

Environment Information :
System : Windows-XP-5.1.2600-SP3
OS Name : nt
Operating System Release : XP
Operating System Version : 5.1.2600
Operating System Architecture : 32bit
Operating System Locale : es_ES.cp1252
Python Version : 2.5.2
OpenERP-Client Version : 5.0.14
Last revision No. & ID :Bazaar Package not Found !Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/openerp-server/netsvc.py", line 299, in dispatch
    result = LocalService(service_name)(method, *params)
  File "/usr/lib/python2.5/site-packages/openerp-server/netsvc.py", line 77, in __call__
    return getattr(self, method)(*params)
  File "/usr/lib/python2.5/site-packages/openerp-server/service/web_services.py", line 577, in execute
    res = service.execute(db, uid, object, method, *args)
  File "/usr/lib/python2.5/site-packages/openerp-server/osv/osv.py", line 58, in wrapper
    return f(self, dbname, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/openerp-server/osv/osv.py", line 119, in execute
    res = pool.execute_cr(cr, uid, obj, method, *args, **kw)
  File "/usr/lib/python2.5/site-packages/openerp-server/osv/osv.py", line 111, in execute_cr
    return getattr(object, method)(cr, uid, *args, **kw)
  File "/usr/lib/python2.5/site-packages/openerp-server/addons/nan_poweremail/poweremail_mailbox.py", line 152, in complete_mail
    self.pool.get('poweremail.core_accounts').get_fullmail(cr, uid, id, context)
  File "/usr/lib/python2.5/site-packages/openerp-server/addons/nan_poweremail/poweremail_core.py", line 998, in get_fullmail
    server_ref, mailid, context)
  File "/usr/lib/python2.5/site-packages/openerp-server/addons/nan_poweremail/poweremail_core.py", line 755, in complete_mail
    parsed_mail = self.get_payloads(mail)
  File "/usr/lib/python2.5/site-packages/openerp-server/addons/nan_poweremail/poweremail_core.py", line 1075, in get_payloads
    parsed_mail['text'] = tools.ustr(part.get_payload(decode=True)) # decode=True to decode a MIME message
  File "/usr/lib/python2.5/site-packages/openerp-server/tools/misc.py", line 746, in ustr
    raise UnicodeError('unable de to convert %r' % (orig,))
UnicodeError: unable de to convert 'Cuatro cosas que debes saber\r\nGmail es un poco diferente. Conoce estos cuatro conceptos b\xe1sicos y ya no\r\nquerr\xe1s usar ninguna otra aplicaci\xf3n.\r\n[image: Inbox screenshot]\r\n\r\n1. Archivar en lugar de eliminar\r\nOrdena "Recibidos" sin eliminar nada. Siempre podr\xe1s realizar una b\xfasqueda\r\npara encontrar lo que necesitas o echar un vistazo a "Todos".\r\n\r\n2. Chat y chat de v\xeddeo\r\nCharla directamente desde Gmail. Incluso puedes hablar cara a cara mediante\r\nel chat de v\xeddeo integrado.\r\n\r\n3. Etiquetas en lugar de carpetas\r\nLas etiquetas funcionan igual que las carpetas con una ventaja adicional:\r\npuedes a\xf1adir m\xe1s de una a un mensaje.\r\n\r\n\r\n[image: Conversation screenshot]\r\n\r\n4. Vista de conversaci\xf3n\r\nGmail agrupa los mensajes y sus respuestas en "Recibidos", de manera que\r\nsiempre se ven los mensajes en el contexto de su conversaci\xf3n. Los mensajes\r\nrelacionados se van colocando ordenadam...

[Omar (Pexego) (omar7r)]

Hi all,

I solved this bug with small patch in tools of openerp-server.
ustr method calls to get_encoding() function, this iterable function returns by default utf-8, it it isn't the correct coding then it returns the locale coding, in Linux case utf-8, if it isn't correct, check if the locale coding it's latin1 or iso-8859-1 or cp1252 and returns an equivalent coding, in Linux case it returns utf-8 too. Then for mails that was sent with Windows, if you try to open its in Linux, power email fails because mail coding is for example so-8859-15 or cp1252 and in linux, it never tests in ustr() method. This issue is solve test always returning latin1 or iso-8859-1 or cp1252 after 'utf-8 and locale coding.

[t.o (emailbox4oerp)]

Hello,

I installed power email today (0.6.3)
OpenERP Server: on revision 2170
OpenERP Client Web: on revision 2951.

Most of the emails are well displayed in the ' Standard Body (Text)' tab but the ' Body (HTML-Web Client Only)' tab shows only 'False'.

The other emails are badly encoded :
- From : =?ISO-8859-1?Q?H=E9l=E8ne_Michel?= <email address hidden>
-subject : =?ISO-8859-1?Q?OpenERP=2C_SC-Assistance_et_TUX_Services_?= =?ISO-8859-1?Q?vous_invitent_=E0_un_s=E9minaire_=E0_Epinal_?=
but in that case the the ' Body (HTML-Web Client Only)' tab shows the entire email body (well displayed) while the ' Standard Body (Text)' tab shows only 'False'