OpenLP crashes when importing songs from EasyWorship

[Jonas Lundkwist]

**OpenLP Bug Report**
Version: {u'full': u'2.0.4', u'version': u'2.0.4', u'build': None}

--- Details of the Exception. ---

When I try to import a songdatabase from EasyWorship 2009, OpenLP crashes after 61% of the database has been imported.

 --- Exception Traceback ---
Traceback (most recent call last):
  File "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.core.ui.wizard", line 190, in onCurrentIdChanged
  File "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.plugins.songs.forms.songimportform", line 391, in performWizard
  File "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.plugins.songs.lib.ewimport", line 191, in doImport
  File "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.plugins.songs.lib", line 596, in strip_rtf
UnicodeDecodeError: 'ascii' codec can't decode byte 0x86 in position 0: ordinal not in range(128)

--- System information ---
Plattform: Windows-7-6.1.7601-SP1 Home Premium
Office version 2010

--- Library Versions ---
Python: 2.7.3
Qt4: 4.8.3
Phonon: 4.4.0
PyQt4: 4.9.5
QtWebkit: 534.34
SQLAlchemy: 0.7.7
SQLAlchemy Migrate: 0.7.2
BeautifulSoup: 3.2.1
lxml: 2.3.0
Chardet: 1.0.1
PyEnchant: 1.6.5
PySQLite: 1.0.1
Mako: 0.7.0
pyUNO bridge: -

[Phill (phill-ridout)]

Hi Jonas,

Could you please email us the database (or email a link to it on DropBox or similar) to <email address hidden>. Please include a link to this page.

[Phill (phill-ridout)]

Sample files linked to from: http://support.openlp.org/scp/tickets.php?id=2302

[Tomas Groth (tomasgroth)]

Tried to import the example files in trunk (2.2), and got this traceback right away:
Traceback (most recent call last):
  File "/home/tgc/OpenLP/openlp/trunk/openlp/core/ui/wizard.py", line 218, in on_current_id_changed
    self.perform_wizard()
  File "/home/tgc/OpenLP/openlp/trunk/openlp/plugins/songs/forms/songimportform.py", line 354, in perform_wizard
    importer.do_import()
  File "/home/tgc/OpenLP/openlp/trunk/openlp/plugins/songs/lib/ewimport.py", line 172, in do_import
    self.title = self.get_field(fi_title).decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe5 in position 11: invalid continuation byte

So it seems we have some issues with EasyWorship DB import...

[Tomas Groth (tomasgroth)]

Hi Jonas

The reason for the crash/traceback is that the titles starting with "SFB" has packed the text in an unexpected way. I've found a way to handle this which fixes the crash, but the SFB titles also uses a different character encoding than the rest of the DB, and this means that the special characters (like å, ö and ä) gets messed up, since the entire DB is expected to use the same encoding.

So I hope you can help me understand how this mixed encoding could happen:
How did you get the SFB titles into the DB? I assume you used some import feature?

[Jonas Lundkwist (jonas-lundkwist)]

Hi Tomas.

SFB stands for "Svenska FolkBibeln" and it is imported by an exe-file
packed by the translators, so I don't know how it is built up. If you want
I can upload the exe-file to.

regards/
Jonas
Den 5 maj 2014 11:05 skrev "Tomas Groth" <email address hidden>:

> Hi Jonas
>
> The reason for the crash/traceback is that the titles starting with
> "SFB" has packed the text in an unexpected way. I've found a way to
> handle this which fixes the crash, but the SFB titles also uses a
> different character encoding than the rest of the DB, and this means
> that the special characters (like å, ö and ä) gets messed up, since the
> entire DB is expected to use the same encoding.
>
> So I hope you can help me understand how this mixed encoding could happen:
> How did you get the SFB titles into the DB? I assume you used some import
> feature?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1299837
>
> Title:
> OpenLP crashes when importing songs from EasyWorship
>
> Status in OpenLP - Worship Presentation Software:
> In Progress
>
> Bug description:
> **OpenLP Bug Report**
> Version: {u'full': u'2.0.4', u'version': u'2.0.4', u'build': None}
>
> --- Details of the Exception. ---
>
> When I try to import a songdatabase from EasyWorship 2009, OpenLP
> crashes after 61% of the database has been imported.
>
> --- Exception Traceback ---
> Traceback (most recent call last):
> File
> "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.core.ui.wizard",
> line 190, in onCurrentIdChanged
> File
> "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.plugins.songs.forms.songimportform",
> line 391, in performWizard
> File
> "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.plugins.songs.lib.ewimport",
> line 191, in doImport
> File
> "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.plugins.songs.lib",
> line 596, in strip_rtf
> UnicodeDecodeError: 'ascii' codec can't decode byte 0x86 in position 0:
> ordinal not in range(128)
>
> --- System information ---
> Plattform: Windows-7-6.1.7601-SP1 Home Premium
> Office version 2010
>
> --- Library Versions ---
> Python: 2.7.3
> Qt4: 4.8.3
> Phonon: 4.4.0
> PyQt4: 4.9.5
> QtWebkit: 534.34
> SQLAlchemy: 0.7.7
> SQLAlchemy Migrate: 0.7.2
> BeautifulSoup: 3.2.1
> lxml: 2.3.0
> Chardet: 1.0.1
> PyEnchant: 1.6.5
> PySQLite: 1.0.1
> Mako: 0.7.0
> pyUNO bridge: -
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openlp/+bug/1299837/+subscriptions
>

[Tomas Groth (tomasgroth)]

Hi Jonas

Thanks for your fast reply! I think I've found the program you mentioned here: http://www.folkbibeln.se/ladda-ner?id=422&iad=110
I think this also explains how the unexpected formatting and encoding got into the DB.
I'm not sure we can handle this in a proper way, so there is a good change the my fix will just chose to skip these "songs" with unexpected formatting and encoding.

[Jonas Lundkwist (jonas-lundkwist)]

Hi again.

Yes, thats the one. I haven't got the time to remove the SFB entries and
retry importing the database. If I recall it right, that bible is already
present in OpenLP, alternatively as an import from OpenSong.

regards/
Jonas
Den 5 maj 2014 13:25 skrev "Tomas Groth" <email address hidden>:

> Hi Jonas
>
> Thanks for your fast reply! I think I've found the program you mentioned
> here: http://www.folkbibeln.se/ladda-ner?id=422&iad=110
> I think this also explains how the unexpected formatting and encoding got
> into the DB.
> I'm not sure we can handle this in a proper way, so there is a good change
> the my fix will just chose to skip these "songs" with unexpected formatting
> and encoding.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1299837
>
> Title:
> OpenLP crashes when importing songs from EasyWorship
>
> Status in OpenLP - Worship Presentation Software:
> In Progress
>
> Bug description:
> **OpenLP Bug Report**
> Version: {u'full': u'2.0.4', u'version': u'2.0.4', u'build': None}
>
> --- Details of the Exception. ---
>
> When I try to import a songdatabase from EasyWorship 2009, OpenLP
> crashes after 61% of the database has been imported.
>
> --- Exception Traceback ---
> Traceback (most recent call last):
> File
> "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.core.ui.wizard",
> line 190, in onCurrentIdChanged
> File
> "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.plugins.songs.forms.songimportform",
> line 391, in performWizard
> File
> "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.plugins.songs.lib.ewimport",
> line 191, in doImport
> File
> "D:\OpenLP_Code\2.0\build\pyi.win32\OpenLP\out00-PYZ.pyz\openlp.plugins.songs.lib",
> line 596, in strip_rtf
> UnicodeDecodeError: 'ascii' codec can't decode byte 0x86 in position 0:
> ordinal not in range(128)
>
> --- System information ---
> Plattform: Windows-7-6.1.7601-SP1 Home Premium
> Office version 2010
>
> --- Library Versions ---
> Python: 2.7.3
> Qt4: 4.8.3
> Phonon: 4.4.0
> PyQt4: 4.9.5
> QtWebkit: 534.34
> SQLAlchemy: 0.7.7
> SQLAlchemy Migrate: 0.7.2
> BeautifulSoup: 3.2.1
> lxml: 2.3.0
> Chardet: 1.0.1
> PyEnchant: 1.6.5
> PySQLite: 1.0.1
> Mako: 0.7.0
> pyUNO bridge: -
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/openlp/+bug/1299837/+subscriptions
>