v6 employee/employees timesheet Dataerror

[GEM]

menus :
human resources>reporting>timesheet>employee timesheet
human resources>reporting>timesheet>employees timesheet

same error for this 2 submenus :

Environment Information :
System : Windows-32bit-SP2
OS Name : nt
Operating System Release :
Operating System Version : 32bit
Operating System Architecture : 32bit
Operating System Locale : fr_FR.cp1252
Python Version : 2.5.4
OpenERP-Client Version : 6.0dev
Last revision No. & ID :Bazaar Package not Found !Traceback (most recent call last):
  File "netsvc.pyo", line 401, in dispatch
  File "service\web_services.pyo", line 584, in dispatch
  File "osv\osv.pyo", line 57, in wrapper
  File "osv\osv.pyo", line 133, in execute
  File "osv\osv.pyo", line 123, in execute_cr
  File "d:\OpenERP AllInOne\Server\addons\base_calendar\base_calendar.py", line 1755, in get
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_values.py", line 221, in get
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_values.py", line 221, in <lambda>
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_values.py", line 207, in _result_get
  File "d:\OpenERP AllInOne\Server\addons\hr\hr_department.py", line 102, in read
  File "osv\orm.pyo", line 2805, in read
  File "osv\orm.pyo", line 2941, in _read_flat
  File "osv\fields.pyo", line 695, in get
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_actions.py", line 172, in _search_view
  File "osv\orm.pyo", line 1136, in fields_get
  File "tools\misc.pyo", line 802, in cached_result
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_translation.py", line 136, in _get_source
  File "sql_db.pyo", line 74, in wrapper
  File "sql_db.pyo", line 119, in execute
DataError: invalid byte sequence for encoding "UTF8": 0xe97672
HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".

[Ravindra Mekhiya(OpenERP) (rme-openerp)]

Hi GEM,

I have tried with most accented characters, but I failed to reproduce the problem.

Would you please send me the characters or the csv file,so I can use the same for this bug?

Thanks.

[GEM (nimp3)]

Hi,
should I export all, or could you say me what are fields to export?
I stand your answer.
Bye

[Ravindra Mekhiya(OpenERP) (rme-openerp)]

Hello GEM,

You can send me only Non-English character which is affected at your end.

Thanks.

[GEM (nimp3)]

Hi RME,
I have the same problem with the english language ...
I made no modification for report.
Are you sure it's a problem of translation ?
I don't know what is the character which give the problem.
Can you give me a direction to follow to find this character ?
Bye

[Ravindra Mekhiya(OpenERP) (rme-openerp)]

Hi GEM,

There seems no error in English language at my end, I can easily print both the reports.

Would you please update the code, recheck ?

This error may occur when Database is not able to handle some values belonging to different encoding.

Thanks.

[GEM (nimp3)]

Hi,
now the error is line 160
Environment Information :
System : Windows-32bit-SP2
OS Name : nt
Operating System Release :
Operating System Version : 32bit
Operating System Architecture : 32bit
Operating System Locale : fr_FR.cp1252
Python Version : 2.5.4
OpenERP-Client Version : 6.0dev
Last revision No. & ID :Bazaar Package not Found !Traceback (most recent call last):
  File "netsvc.pyo", line 435, in dispatch
  File "service\web_services.pyo", line 586, in dispatch
  File "d:\OpenERP AllInOne\Server\addons\audittrail\audittrail.py", line 503, in execute
  File "d:\OpenERP AllInOne\Server\addons\audittrail\audittrail.py", line 488, in my_fct
  File "osv\osv.pyo", line 57, in wrapper
  File "osv\osv.pyo", line 135, in execute
  File "osv\osv.pyo", line 125, in execute_cr
  File "d:\OpenERP AllInOne\Server\addons\base_calendar\base_calendar.py", line 1772, in get
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_values.py", line 221, in get
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_values.py", line 221, in <lambda>
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_values.py", line 207, in _result_get
  File "d:\OpenERP AllInOne\Server\addons\hr\hr_department.py", line 98, in read
  File "osv\orm.pyo", line 2976, in read
  File "osv\orm.pyo", line 3127, in _read_flat
  File "osv\fields.pyo", line 697, in get
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_actions.py", line 172, in _search_view
  File "osv\orm.pyo", line 1212, in fields_get
  File "tools\misc.pyo", line 809, in cached_result
  File "d:\OpenERP AllInOne\Server\addons\base\ir\ir_translation.py", line 160, in _get_source
  File "sql_db.pyo", line 75, in wrapper
  File "sql_db.pyo", line 124, in execute
DataError: invalid byte sequence for encoding "UTF8": 0xe97672
HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".

but solved with this change :
-(lang or '', tt, tools.ustr(name), source))
+(lang or '', tt, tools.ustr(name), tools.ustr(source)))

Bye

[GEM (nimp3)]

Hi,
bug can be closed, i have any more this error
Bye

[Jay Vora (Serpent Consulting Services) (jayvora)]

Thanks.

[Laurent Urbain (Realdolmen) (laurent-urbain)]

Hi,

I have this bug with the last trunk version of addons and server on a windows server.

Do you have the same problem again or it comes from windows or other?

[GEM (nimp3)]

hi,
I can't test because with last sources I build as installer, I can't create a database.
do you use only sources without build installer ?
else I have no more this eror since I uninstall mx datetime 3.1.1 from python librarries and install the last version 3.1.3
http://www.egenix.com/products/python/mxBase/
hope this will solve your problem.
Bye

[GEM (nimp3)]

another change for library required is python-dateutil to install : easy_install python-dateutil (setuptools must be installed to launch the command in console mode)
I install my build, and corrected the error I had after installation, I created a database profile with all options checked without error using configurator :
server 3058
addons 3977

[Laurent Urbain (Realdolmen) (laurent-urbain)]

Yes, I use only sources without build installer

I installed the last version of mx datetime (3.1.3) and python-dateutil.

But I have yet the error when I want print employee(s) timesheet :

Traceback (most recent call last):
  File "E:\OpenERP\trunk\server\bin\netsvc.py", line 489, in dispatch
    result = ExportService.getService(service_name).dispatch(method, auth, params)
  File "E:\OpenERP\trunk\server\bin\service\web_services.py", line 596, in dispatch
    res = fn(db, uid, *params)
  File "E:\OpenERP\trunk\server\bin\osv\osv.py", line 119, in wrapper
    return f(self, dbname, *args, **kwargs)
  File "E:\OpenERP\trunk\server\bin\osv\osv.py", line 203, in execute
    res = pool.execute_cr(cr, uid, obj, method, *args, **kw)
  File "E:\OpenERP\trunk\server\bin\osv\osv.py", line 193, in execute_cr
    return getattr(object, method)(cr, uid, *args, **kw)
  File "E:\OpenERP\trunk\server\bin\osv\orm.py", line 1748, in fields_view_get
    xarch, xfields = self.__view_look_dom_arch(cr, user, result['arch'], view_id, context=context)
  File "E:\OpenERP\trunk\server\bin\osv\orm.py", line 1453, in __view_look_dom_arch
    fields = self.fields_get(cr, user, fields_def.keys(), context)
  File "E:\OpenERP\trunk\server\bin\osv\orm.py", line 1256, in fields_get
    val2 = translation_obj._get_source(cr, user, self._name + ',' + f, 'selection', context.get('lang', False) or 'en_US', val)
  File "E:\OpenERP\trunk\server\bin\tools\misc.py", line 797, in cached_result
    result2 = fn(self2, cr, *args[:self.skiparg-2], **kwargs2)
  File "E:\OpenERP\trunk\server\bin\addons\base\ir\ir_translation.py", line 162, in _get_source
    (lang or '', types, tools.ustr(name), source))
  File "E:\OpenERP\trunk\server\bin\sql_db.py", line 76, in wrapper
    return f(self, *args, **kwargs)
  File "E:\OpenERP\trunk\server\bin\sql_db.py", line 129, in execute
    res = self._obj.execute(query, params)
DataError: invalid byte sequence for encoding "UTF8": 0xe97672
HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".

In which file I must do this change :

-(lang or '', tt, tools.ustr(name), source))
+(lang or '', tt, tools.ustr(name), tools.ustr(source)))

??

Thx for your help!

[GEM (nimp3)]

Hi,
it is possible that even if you update the code with bazaar, files .pyo are not overwritten (this is files which are used by openerp).
to exit this suggestion, try to delete all files .pyo in module used where you have the error, restart the server.
else if againt the error, delete all files in server\*, and restart the server.
else save actual folder server and download a new install of server and try.
good test

[Laurent Urbain (Realdolmen) (laurent-urbain)]

Hi,

I deleted all the .pyc and restart, I have the error yet.

I downloaded all the last server sources and restart the server from this sources.
I have yet the problem.

if I install the server from openerp-server-setup-6.0.0-rc1.exe it can change anything?
It's only the server in the .exe or there is too postgresl python dependencies,...?

thx!

[GEM (nimp3)]

oups,
perhaps try this, forget to say you to update the module after deleted files .pyc or .pyo, backup the database and server folder first, cause to valid update of sources you must restart and update modules

did you have installed postgresql in utf8 encoding ?

try to create a new database too and test (after deleting .pyc .pyo, restart, update modules)

try to use my modification in server\bin\addons\base\ir\ir_translation.py" (delete file .pyc .pyo, restart server)

-(lang or '', tt, tools.ustr(name), source))
+(lang or '', tt, tools.ustr(name), tools.ustr(source)))

try to do a complete fresh install and new database (backup what you do before, folders and database)

if you want to use installer for the server, save your openerp-server.conf, install, overwrite the conf file restart server (no need to install postgresql and dependencies)

[GEM (nimp3)]

you can try to import your database backup online with last sources (no need to reinstall all), just connect this url and restore your database you backuped:
http://revision6.openerp-online.com/
thanks to Bounaberdi (openerp forum) from sisalp to give this contribution for free

[JMA(Open ERP) (jma-openerp)]

Hello Guys,

Just made an instant test but there seem nothing to break.
@laurent: may be the .pyo or .pyc files need to be updated. Check with a fresh installation may be that could help you.

Would like to meet the error.
Thanks.

[GEM (nimp3)]

Hi jMA,
personnaly for me there is no problem since I done a fresh install. I think also it is a problem of pyo or update, some paths and files name have changed too.
I believe it could be better that laurentO ocretaes a topic on the forum for this, where I could follow this.
Bye

[Laurent Urbain (Realdolmen) (laurent-urbain)]

Hi,

I attend an OpenERP training until tomorrow, I will look on it on friday.

Thx for your help

[Laurent Urbain (Realdolmen) (laurent-urbain)]

Hi

I deleted all the pyo and pyc on the server.
I Downloaded the last sources
I modified in server\bin\addons\base\ir\ir_translation.py
Create a new db in UTF8...and it works.

But if in the project's name, there is a é, è, à,....
I have an ascii error :
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1831 : ordinal not in range(128)

[GEM (nimp3)]

Hi,
can you give the name(s) of field(s) in sources where you have the error ?
bye

[Laurent Urbain (Realdolmen) (laurent-urbain)]

It's the same problem with each fields in the timesheet.

e.g. If there's an é in the employee's name it's the same problem.

Do you want I create a topic on the forum?
Thx

Laurent

[GEM (nimp3)]

Hi,
I believe that str() shouldn't be used with error that this function give always as you have :
UnicodeDecodeError

see and change in (save the original file):

server\addons\hr\hr_department.py line 98 modify like thisfollowing -/+ :
- r['domain'] = r['domain'].replace(mystring, str(obj_dept._get_members(cr, uid)))
+ r['domain'] = r['domain'].replace(mystring, tools.ustr(obj_dept._get_members(cr, uid)))
 (ustr is utf8 encoding)

delete files :
hr_department.pyc
hr_department.pyo
restart the server
update the module

I believe it is a possibility (but a recurrent error in openerp code), but it should be an other thing.
Try and test.
thanks
bye

[Laurent Urbain (Realdolmen) (laurent-urbain)]

Yes same error with this change in hr_department.py.

In the last trunk, there are modifications in ir_translation.py
So I suppose the line to change now is the 161 :

 - params = (lang or '', types, source)
+ params = (lang or '', types, tools.ustr(source))

[Olivier Dony (Odoo) (odo-openerp)]

Basically, using strftime("%B") or any of the local-dependent values (%a,%b,%c,etc.) is unsafe because there are platforms where the default locale will not be using Unicode charset. This can be clearly seen in the bug description, where the locale is 'fr_FR.cp1252' i.e. french with Windows-1252 charset, Microsoft's variation of Latin1.
And indeed, the 0xe97672 bytes mentioned in the error represent "évr" in Windows-1252. If you need to convince yourself:
  >>> "\xe9\x76\x72".decode('latin1') == u"évr"
  True

With a UTF8 locale this behaves correctly, even with accents:
  >>> import datetime
  >>> locale.setlocale(locale.LC_ALL,'')
  'fr_BE.utf8'
  >>> datetime.datetime(2000,2,1).strftime('%B')
  'f\xc3\xa9vrier' # this is UTF-8 for 'février'

I'm afraid we can't rely on the default locale for translations, as it can fail but will also be wrong in many cases, and we can't force the locale to 'en_US' because locales are system-dependent and it may not be available.

A cleaner solution would be to have a map of english month names in tools.misc, with a simple lookup function that all modules can depend on, and replace all references to strftime('%B') etc. by a lookup in this map, followed by a call to _( ) to properly translate it to the user language.
However this requires changing many modules.

We can already mitigate this issue by making sure the terms to be translated in ir.translation.get_source() are converted to valid UTF-8 first, to catch the DataError cases. The translation may still not work, because there will be no translation for non-english month names, but at least the user will see the month/day names in the locale of the server, better than nothing.

[Olivier Dony (Odoo) (odo-openerp)]

The workaround for ir.translation has landed in revision 3193 <email address hidden>/
Thanks for reporting!