problems on titles with apostrophes

[Castle085]

Entity: line 63: parser error : Opening and ending tag mismatch: module line 18 and pr
</pr></rojo><br />
     ^
Entity: line 63: parser error : Opening and ending tag mismatch: pr line 17 and rojo
</pr></rojo><br />
            ^
Entity: line 67: parser error : Opening and ending tag mismatch: rojo line 17 and margin12
            </margin12>
                       ^
Entity: line 68: parser error : Opening and ending tag mismatch: margin12 line 13 and xhtml
        </xhtml>
                ^
Entity: line 69: parser error : Premature end of data in tag xhtml line 2

        ^
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/share/wiithon/gui.py", line 397, in excepthook
    self.alert('error' , mensaje_xml, excvalue, xml=True)
  File "/usr/share/wiithon/gui.py", line 900, in alert
    view.render_xml(xml_plantilla)
  File "/usr/share/wiithon/textview_custom.py", line 95, in render_xml
    doc = libxml2.parseDoc(xml)
  File "/usr/lib/pymodules/python2.6/libxml2.py", line 1263, in parseDoc
    if ret is None:raise parserError('xmlParseDoc() failed')
libxml2.parserError: xmlParseDoc() failed

Original exception was:
Traceback (most recent call last):
  File "/usr/games/wiithon", line 91, in <module>
    App()
  File "/usr/games/wiithon", line 75, in App
    interfaz = WiithonGUI(core)
  File "/usr/share/wiithon/gui.py", line 275, in __init__
    self.refrescarParticionesWBFS()
  File "/usr/share/wiithon/gui.py", line 307, in refrescarParticionesWBFS
    self.leer_juegos_de_las_particiones(self.lParti)
  File "/usr/share/wiithon/gui.py", line 986, in leer_juegos_de_las_particiones
    self.core.syncronizarJuegos(particion)
  File "/usr/share/wiithon/core.py", line 37, in syncronizarJuegos
    juego = session.query(Juego).filter(sql).first()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/query.py", line 1231, in first
    ret = list(self[0:1])
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/query.py", line 1152, in __getitem__
    return list(res)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/query.py", line 1291, in __iter__
    self.session._autoflush()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/session.py", line 899, in _autoflush
    self.flush()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/session.py", line 1354, in flush
    self._flush(objects)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/session.py", line 1432, in _flush
    flush_context.execute()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/unitofwork.py", line 257, in execute
    UOWExecutor().execute(self, tasks)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/unitofwork.py", line 720, in execute
    self.execute_save_steps(trans, task)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/unitofwork.py", line 735, in execute_save_steps
    self.save_objects(trans, task)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/unitofwork.py", line 726, in save_objects
    task.mapper._save_obj(task.polymorphic_tosave_objects, trans)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/orm/mapper.py", line 1387, in _save_obj
    c = connection.execute(statement.values(value_params), params)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 824, in execute
    return Connection.executors[c](self, object, multiparams, params)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 874, in _execute_clauseelement
    return self.__execute_context(context)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 896, in __execute_context
    self._cursor_execute(context.cursor, context.statement, context.parameters[0], context=context)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 950, in _cursor_execute
    self._handle_dbapi_exception(e, statement, parameters, cursor, context)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 931, in _handle_dbapi_exception
    raise exc.DBAPIError.instance(statement, parameters, e, connection_invalidated=is_disconnect)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'INSERT INTO juego (idgame, title, size, "idParticion") VALUES (?, ?, ?, ?)' [u'RWSP8P', 'Mario y Sonic en los Juegos Ol\xedmpicos', 2.109375, 1]

[Carlo Mandelli (camandel)]

What version-revision are you using?
Have you already tried lastest stable or development release?

1) if you are using the stable version you must describe all the steps to follow to repreduce the issue

2) if you are using a devel version >= 1.21-r423 run this command:

/usr/share/wiithon/wiithon_debug.sh

then execute all the steps to reproduce the issue and then attach to this bug the file created (/tmp/wiithon_debug.log).

[Castle085 (joel-castillo-martinez)]

Uso la version estable. 1.1 sacada de los repositorios para Ubuntu 9.10. ejecuto en una consola el comando wiithon sin HDD y funciona correctamente pero con el HDD aparece el mensaje que poste. Gracias de ante mano.

[Carlo Mandelli (camandel)]

Try this procedure:

1) unplug HDD
2) preferencies --> tools and utilities --> format DB
3) plug HDD
4) import an ISO image

[_tus_ (guillaume-artus)]

This procedure does not work either.

In fact, I reproduced the issue on my own drive using the french name for RY3P41. This name contains some accents:
"Rayman Prod pr\xe9sente: The Lapins Cr\xe9tins Show" and sqlalchemy complains about this letters when the wbfs is read to fill the bdd.

As stated by sqlalchemy: "You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings."

Could it be possible to do this setting? or is it a solution to update the wbfs located iso name to remove the "dealy" letters?

[_tus_ (guillaume-artus)]

Some more information:
- both the 1.1 and the trunk last revision are "faulty" (tested yesterday evening CET time)
- wiithon was able to store the title (obtained through wiitdb) on the wbfs but not in the bdd

[_tus_ (guillaume-artus)]

I investigated deeper the issue:
wiithon_wrapper.c returns game name using the system locale (which may no be utf8 compatible) and this locale is not checked by wiithon.

I provided a quick patch using the hachoir library.

[_tus_ (guillaume-artus)]

full patch provided (including the debian/control)

[Carlo Mandelli (camandel)]

Thank you very much _tus_ for your patch.
Committed on:

1.16-r395 (stable)
1.21-r436 (devel)

[_tus_ (guillaume-artus)]

Improved patch (test if a charset is found, if ok, then use it: solve the RY3P41 issue with french title)

DOES NOT correct the issue with malformated title like:
SBDP08 french title for which the detected charset is none

Not that the patch did not introduce the last issue, this issue was already existing

[Carlo Mandelli (camandel)]

Please, test last devel release (r437)

[Carlo Mandelli (camandel)]

Work in progress....

Please, test last devel version.

[_tus_ (guillaume-artus)]

Performed some check on version 440
still got an issue on SBDP08
with the french title:
(ProgrammingError) You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings. u'INSERT INTO juego (idgame, title, size, "idParticion") VALUES (?, ?, ?, ?)' [u'SBDP08', 'Resident Evil:\xa0The Darkside Chronicles', 3.7304689999999998, 1]

No problem with the UK title :'Resident Evil: The Darkside Chronicles'
The issue is that even guessBytesCharset does not handle the \xa0 character and return None as the found Charset (which blown up the next decode command)

Should we decide to use some default name when the guessBytesCharset return None and to inform the user that the read name is not correct for wiithon (in the same time, checking the charset of the wiitdb title field before written it on the wbfs could be a smart idea, possibly with a warning message asking for a new title)

[_tus_ (guillaume-artus)]

Workaround for people facing the same issue:

use wiithon_wrapper by hand:

"wiithon_wrapper -p /dev/sdc1 rename SBDP08 resident"

allows to reopen wiithon and to select a new name (hopefully a correct one)

[_tus_ (guillaume-artus)]

Investigate new lead:
rework the decode function as in hachoir: try to decode strict ASCI then ISO-8859-1 and finaly fall back to ISO-8859-15
remove the hachoir dependency (both in core.py and in debian/control)

This patch solved both the RY3P41 issue (&eacute; in the french title name) and the SBDP08 one (non breakable space "&nbsp;" in the game tile) on my computer.

Some rework from skilled python developper can be a smart idea ;-)

[Carlo Mandelli (camandel)]

Patch provided by _tus_ committed in:

1.16-r396 (stable)
1.21-r441 (devel)