gnome-dvb-control doesn't display EPG with correct encoding

[Frédéric Crozat]

when running in non-UTF8 locale (fr_FR.ISO8859-15), program names is not converted properly before being set to GTK for display.

For instance, Météo is displayed as MØtØo etc...

[Sebastian Pölsterl (sebp)]

Doesn't GTK+ use utf8 encoded strings all the time?

[Frédéric Crozat (fcrozat)]

yes, gtk+ requires all strings it gets to be UTF-8.

So it means either EPG or something in the decoding process is using "local" encoding instead of forcing UTF-8 (an usual mistake in many gnome apps..). It will work most of the time, since most of the time, local encoding will be UTF-8

[Sebastian Pölsterl (sebp)]

If you run the daemon with "GST_DEBUG=*:2 gnome-dvb-daemon" does it complain that encoding failed (or just attach the output)?

[Frédéric Crozat (fcrozat)]

no, there is absolutely no warning regarding encoding or something like that. I'm getting some "CRITICAL ** : EPGScanner.vala:252 could not find channel xxxx for this device" and some "EPGScanner.vala:214: could not read from DVB device" but nothing more.

[Sebastian Pölsterl (sebp)]

All string should be converted to utf8 inside the gstreamer element. The only thing I can think of right now would be that the encoding used in the transport stream is not detected correctly.

Run the daemon with GST_DEBUG=mpeg*:4. This creates lots of output, but it should contain lines containing "Found encoding xyz". If that's the cause it would be weird that converting to utf8 is successful despite the wrong input encoding, though.

Do you know if other DVB applications display the names correctly?

[Frédéric Crozat (fcrozat)]

finding "working" EPG DVB applications on linux seems to be a hard subject in itself. I've only been able to get vlc to display "program name" but it displays it in the application window title (and not correctly), so I'm not 100% sure it is vlc fault.

I've renamed the bug since I have the issue also when switching to UTF-8 locale so it is a encoding conversion issue in either gnome-dvb-daemon or gstreamer.

Anyway, I got some debug ouput, from two different channels, one in iso6937, one in iso8859-9

First one description should be "Magazine d'actualité" but is displayed in UI : "Magazine d'actualitØ"

Second one description should be : "Présenté par Ruth Elkrief" (I don't have the text which was displayed in the UI)

0:00:12.957906151 19750 0xb6404600 DEBUG mpegtspacketizer mpegtspacketizer
.c:2332:get_encoding: Found encoding iso6937, first byte is 0x4d, start_text: 0,
 is_multibyte: 0
0:00:12.958033409 19750 0xb6404600 DEBUG mpegtsdesc gstmpegdesc.c:17
0:gst_mpeg_descriptor_find_all: found tag 0x4e 0 times
0:00:12.958090742 19750 0xb6404600 DEBUG mpegtsdesc gstmpegdesc.c:17
0:gst_mpeg_descriptor_find_all: found tag 0x50 2 times
0:00:12.958353216 19750 0xb6404600 DEBUG mpegtspacketizer mpegtspacketizer.c:1930:mpegts_packetizer_parse_eit: EIT eit, version-number=(guint)18, current-next-indicator=(guint)1, service-id=(guint)261, actual-transport-stream=(boolean)true, present-following=(boolean)true, transport-stream-id=(guint)1, original-network-id=(guint)8442, segment-last-section-number=(guint)1, last-table-id=(guint)78, events=(structure){ "event-36384\,\ event-id\=\(guint\)36384\,\ year\=\(guint\)2010\,\ month\=\(guint\)4\,\ day\=\(guint\)12\,\ hour\=\(guint\)10\,\ minute\=\(guint\)20\,\ second\=\(guint\)0\,\ duration\=\(guint\)1500\,\ running-status\=\(guint\)1\,\ free-ca-mode\=\(boolean\)false\,\ name\=\(string\)\"Giordano\\\ hebdo\"\,\ description\=\(string\)\"Magazine\\\ d\\\'actualit\\303\\230.\"\,\ components\=\(structure\)\{\ \"audio\\\,\\\ type\\\=\\\(string\\\)stereo\\\,\\\ tag\\\=\\\(int\\\)2\\\;\"\,\ \"video\\\,\\\ high-definition\\\=\\\(boolean\\\)true\\\,\\\ frequency\\\=\\\(int\\\)25\\\,\\\ tag\\\=\\\(int\\\)1\\\,\\\ aspect-ratio\\\=\\\(string\\\)4:3\\\;\"\ \}\,\ descriptors\=\(GValueArray\)NULL\;" };

or

0:00:12.966843362 19750 0xb6404600 DEBUG mpegtspacketizer mpegtspacketizer
.c:2332:get_encoding: Found encoding iso8859-9, first byte is 0x05, start_text: 1, is_multibyte: 0
0:00:12.967014435 19750 0xb6404600 DEBUG mpegtsdesc gstmpegdesc.c:170:gst_mpeg_descriptor_find_all: found tag 0x4e 0 times
0:00:12.967072738 19750 0xb6404600 DEBUG mpegtsdesc gstmpegdesc.c:170:gst_mpeg_descriptor_find_all: found tag 0x50 2 times
0:00:12.967303121 19750 0xb6404600 DEBUG mpegtspacketizer mpegtspacketizer.c:1930:mpegts_packetizer_parse_eit: EIT eit, version-number=(guint)16, current-next-indicator=(guint)1, service-id=(guint)515, actual-transport-stream=(boolean)false, present-following=(boolean)true, transport-stream-id=(guint)2, original-network-id=(guint)8442, segment-last-section-number=(guint)1, last-table-id=(guint)79, events=(structure){ "event-20711\,\ even...

[Sebastian Pölsterl (sebp)]

Is the second string encoded in iso8859-9 converted correctly or is it wrong as well?

[Frédéric Crozat (fcrozat)]

I wasn't sure about the second string, so I did another capture. The following capture is correctly displayed in gnome-dvb-control.

So, it would appear iso8859-9 encoded strings are correctly displayed, but iso6937 aren't. I've checked various channels encoded in iso8859-9 and they are displayed correctly (unfortunately, when using fr-Paris to tune, I was only getting one transponder and all channels on this transponder are using iso6937

log:

0:02:02.448532623 32495 0x98dfbd8 DEBUG mpegtspacketizer mpegtspacketizer
.c:2332:get_encoding: Found encoding iso8859-9, first byte is 0x05, start_text:
1, is_multibyte: 0
0:02:02.449361397 32495 0x98dfbd8 DEBUG mpegtsdesc gstmpegdesc.c:17
0:gst_mpeg_descriptor_find_all: found tag 0x4e 0 times
0:02:02.449555068 32495 0x98dfbd8 DEBUG mpegtsdesc gstmpegdesc.c:17
0:gst_mpeg_descriptor_find_all: found tag 0x50 4 times
RTSP request message 0x98b3200
0:02:02.449956894 32495 0x98dfbd8 DEBUG mpegtspacketizer mpegtspacketizer.c:1930:mpegts_packetizer_parse_eit: EIT eit, version-number=(guint)21, current-next-indicator=(guint)1, service-id=(guint)770, actual-transport-stream=(boolean)true, present-following=(boolean)true, transport-stream-id=(guint)3, original-network-id=(guint)8442, segment-last-section-number=(guint)1, last-table-id=(guint)78, events=(structure){ "event-28\,\ event-id\=\(guint\)28\,\ year\=\(guint\)2010\,\ month\=\(guint\)4\,\ day\=\(guint\)12\,\ hour\=\(guint\)14\,\ minute\=\(guint\)56\,\ second\=\(guint\)0\,\ duration\=\(guint\)6180\,\ running-status\=\(guint\)4\,\ free-ca-mode\=\(boolean\)true\,\ name\=\(string\)\"Les\\\ soldats\\\ du\\\ d\\303\\251sert\"\,\ description\=\(string\)\"R\\303\\251alis\\303\\251\\\ par\\\ Irwin\\\ Winkler\\\ en\\\ 2006.\\\ Avec\\\ Samuel\\\ L\\\ Jackson\\\,\\\ Jessica\\\ Biel\\\,\\\ Brian\\\ Presley.\\\ Drame\\\ de\\\ guerre\\\ am\\303\\251ricain.\\\ \"\,\ components\=\(structure\)\{\ \"audio\\\,\\\ type\\\=\\\(string\\\)stereo\\\,\\\ tag\\\=\\\(int\\\)1\\\;\"\,\ \"audio\\\,\\\ type\\\=\\\(string\\\)stereo\\\,\\\ tag\\\=\\\(int\\\)1\\\;\"\,\ \"video\\\,\\\ high-definition\\

[Sebastian Pölsterl (sebp)]

Apparently broadcaster don't take the spec too seriously, it's a nightmare. The current workaround is to try to decode iso6937 and if that fails try iso8859-9. But in your case the string can be decoded, so we have to find a better way to detect which encoding was actually meant.

[Frédéric Crozat (fcrozat)]

from what I found on vlc mailing list, it is really a nightmare. I have no idea how hardware DVB-T vendors are handling this kind of issues..