wrong charcter set in rtf to mobi conversation

[Alex Samorukov]

I have rtf document which is not correctly converted from RTF to MOBI format, seems that charset is wrong and result is unreadable. This document was created by OpenOffice from the Ms-word format. Its opening correctly in the OpenOffice and Wordpad, so i believe that bug is in Calibre. May be /lang is not parsed correctly? I see that it using lang1049 for the text, what means "russian", so input encoding needs to be cp1251. I`m attaching this document to the ticket. Tell me if you need anything else.

[Alex Samorukov (samm-os2)]

One more comment - adding ansicpg1251 to the rtf header fix detection

[Alex Samorukov (samm-os2)]

Ok, so the problem is that codepage in this document is specified in the \fcharsetN header in the font description. According to spec (http://latex2rtf.sourceforge.net/rtfspec_6.html) 204 is Russian (eq. 1251). This is respected by OpenOffice and Wordpad and probably ignored by the calibre.

[Kovid Goyal (kovid)]

Changing the component for this bug.

 assignee sengian
 tag rtf-input
 status triaged

[Kovid Goyal (kovid)]

Note that if you are using OpenOffice, save your documents in odt and convert that in calibre.

[sengian (sengian)]

I have looked into the linked bug with diacritics, and I am still not sure about it, but concerning this one you are absolutely correct.
The problem is with the tool used by calibre which does not support \fcharsetN. It relies on the \ansicpgN to get the codepage.

[Alex Samorukov (samm-os2)]

Why it is set to invalid? It is 100% bug. This document was produced by OpenOffice - one of the most popular office suites. It is opened correctly both in OO and MS wordpad, so document itself is correct.

My recommendation is to look on ansicpg and if it is not found - try to "guess" charset from the fcharset in the font list.

And thank you for recommendation to use odt, i`ll try this.

[sengian (sengian)]

I agree, I will put it to won't wix if you want as it is on the TODO list but currently a limitation of the RTF conversion tool. The problem is with the guess you propose is that if you look, you will see that there is a lot of fonts being declared.
FYI, the ansicpg is declared but is superseeded by the \fcharset, so I need to implement this in the tool and in the conversion.

[Alex Samorukov (samm-os2)]

Yes, i understand that this is not real solution, because its possible to define more than one \fcharset for different font (anyone using this?), so probably implementation needs to be rewritten to use encoding per object and not only per document. So thank you for reply and explanation. Unfortunately i am not a python developer, so can`t help with that.

P.S. conversion from ODT works fine on this document.

[Kovid Goyal (kovid)]

@sengian: Are you actually planning to implement support for fcharset in rtf2xml? If not, feel free to close this ticket as wontfix. Personally, I have no interest in supporting fcharset in RTF.

[sengian (sengian)]

I have, but as I have no idea of how long it will take, so I will close it for know.