Profile view, Edit: freeze of the UI when adding the external video file block using a wrongly encoded L10N

[Matteo]

Mahara 1.2.2 (vanilla)
LAMP, PHP 5.2.12
MySQL
Integrated with Moodle, SSO IN (Moodle > Mahara)

It seems just related toa missing configuration parameter, ''videoid''; is it a bug or an error/missing done during the installation? IMHO no reason for the JSON encode error since no data has been passed yet and no chance at all to break any enconding sequence.

Find below the complete log around the action described in the summary:
{{{
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] [WAR] 79 (blocktype/externalvideo/lib.php:61) Undefined index: videoid, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] Call stack (most recent first):, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * log_message("Undefined index: videoid", 8, true, true, "/path/to/mahara-stage/mahara-1.2.2/blocktype/exter...", 61) at /path/to/mahara-stage/mahara-1.2.2/lib/errors.php:446, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * error(8, "Undefined index: videoid", "/path/to/mahara-stage/mahara-1.2.2/blocktype/exter...", 61, array(size 4)) at /path/to/mahara-stage/mahara-1.2.2/blocktype/externalvideo/lib.php:61, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * PluginBlocktypeExternalvideo::render_instance(object(BlockInstance), true) at Unknown:0, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * call_user_func_array(array(size 2), array(size 2)) at /path/to/mahara-stage/mahara-1.2.2/lib/mahara.php:1030, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * call_static_method("PluginBlocktypeExternalvideo", "render_instance", object(BlockInstance), true) at /path/to/mahara-stage/mahara-1.2.2/blocktype/lib.php:519, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * BlockInstance->render_editing(false, true) at /path/to/mahara-stage/mahara-1.2.2/lib/view.php:1220, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * View->addblocktype(array(size 4)) at /path/to/mahara-stage/mahara-1.2.2/lib/view.php:944, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * View->process_changes() at /path/to/mahara-stage/mahara-1.2.2/view/blocks.json.php:46, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] , referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] [WAR] 79 (blocktype/externalvideo/lib.php:92) Undefined index: videoid, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] Call stack (most recent first):, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * log_message("Undefined index: videoid", 8, true, true, "/path/to/mahara-stage/mahara-1.2.2/blocktype/exter...", 92) at /path/to/mahara-stage/mahara-1.2.2/lib/errors.php:446, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * error(8, "Undefined index: videoid", "/path/to/mahara-stage/mahara-1.2.2/blocktype/exter...", 92, array(size 2)) at /path/to/mahara-stage/mahara-1.2.2/blocktype/externalvideo/lib.php:92, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * PluginBlocktypeExternalvideo::instance_config_form(object(BlockInstance), "0") at Unknown:0, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * call_user_func_array(array(size 2), array(size 2)) at /path/to/mahara-stage/mahara-1.2.2/lib/mahara.php:1030, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * call_static_method("PluginBlocktypeExternalvideo", "instance_config_form", object(BlockInstance), "0") at /path/to/mahara-stage/mahara-1.2.2/blocktype/lib.php:652, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * BlockInstance->build_configure_form(true) at /path/to/mahara-stage/mahara-1.2.2/blocktype/lib.php:515, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * BlockInstance->render_editing(true, true) at /path/to/mahara-stage/mahara-1.2.2/lib/view.php:1223, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * View->addblocktype(array(size 4)) at /path/to/mahara-stage/mahara-1.2.2/lib/view.php:944, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * View->process_changes() at /path/to/mahara-stage/mahara-1.2.2/view/blocks.json.php:46, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] , referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] [WAR] 79 (lib/web.php:909) json_encode() [function.json-encode]: Invalid UTF-8 sequence in argument, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] Call stack (most recent first):, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * log_message("json_encode() [function.json-encode]: Invalid UTF-...", 8, true, true, "/path/to/mahara-stage/mahara-1.2.2/lib/web.php", 909) at /path/to/mahara-stage/mahara-1.2.2/lib/errors.php:446, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * error(2, "json_encode() [<a href='function.json-encode'>func...", "/path/to/mahara-stage/mahara-1.2.2/lib/web.php", 909, array(size 3)) at Unknown:0, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * json_encode(array(size 3)) at /path/to/mahara-stage/mahara-1.2.2/lib/web.php:909, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] * json_reply(false, array(size 2)) at /path/to/mahara-stage/mahara-1.2.2/view/blocks.json.php:47, referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
[Wed Jan 27 16:32:39 2010] [error] [client 12.34.56.78] , referer: http://111.222.333.444/mahara-stage/view/blocks.php?id=15&c=fileimagevideo&new=
}}}

TIA,
Matteo

[Richard Mansfield (richard-mansfield)]

I have removed the warning about the undefined 'videoid', but I don't think that will fix the problem. I can't reproduce the UI freeze yet.

Does it happen as soon as you drag the external video block into your view? Which browser are you using?

[Matteo (matteo-scaramuccia)]

1. Yes, as soon as I drag the block into the view. A page refresh (F5) will show the block in the view and the freeze re-appears on editing that block in the view, while block deletion works as expected
2. FF 3.5.7, IE 8.0.

Can the JSON error affect the issue too? I can try to change the Apache conf (now: AddDefaultCharset Off), if needed.

TIA

[Richard Mansfield (richard-mansfield)]

Hi Matteo,
I can't work out where the JSON error is coming from. It doesn't happen on my installation. It's complaining about a non-UTF8 sequence, so I guess one possibility is that it's a language string with a non-UTF8 character in it. Are you using a language pack? If so, does the error still happen when you switch to English?

[Matteo (matteo-scaramuccia)]

Hi Richard,
you've found the reason!
Switching to English has solved the issue but, since we're interested in using mainly Italian, we're currently using the pack found here: http://wiki.mahara.org/Download_Mahara.
The issue is around "è" being not saved as a valid UTF-8 char (no mb if you look at it using an Hex view):

# diff -u blocktype.externalvideo.php.orig blocktype.externalvideo.php
--- blocktype.externalvideo.php.orig 2010-01-29 14:24:15.000000000 +0100
+++ blocktype.externalvideo.php 2010-01-29 14:27:10.000000000 +0100
@@ -29,8 +29,8 @@
 $string['title'] = 'Video esterni';
 $string['description'] = 'Video esterni incorporati';
 $string['videourl'] = 'URL del Video';
-$string['videourldescription2'] = 'Incolla il codice incorporato o l\'URL della pagina dove il Video è localizzato. Puoi aggiungere video incorporati dai seguenti siti:';
+$string['videourldescription2'] = 'Incolla il codice incorporato o l\'URL della pagina dove il Video e\' localizzato. Puoi aggiungere video incorporati dai seguenti siti:';
 $string['width'] = 'Larghezza';
 $string['height'] = 'Altezza';

-?>
\ No newline at end of file
+?>

e.g. using an ASCII representation of the accent ("é" -> "e'") solves the issue.

Questions:
1. Is it possible to avoid such blocking issues for any fault from "L10N" even encoding ones e.g. catching some way the exception? Maybe using a combination of prepending "@" and some check after the encode...

2. Is it possible to quickly validate the UTF-8 encoding correctness of a language pack and fix the current set of language packs? I.e possible check through iconv comand:

# iconv -f UTF-8 blocktype.externalvideo.php.orig -o /dev/null
iconv: illegal input sequence at position 1328

So, supposing the file to be wrongly encoded using ISO-8859-1 (è<->hex E8), this could be a way of fixing the IT file

# iconv -f ISO-8859-1 blocktype.externalvideo.php.orig -o blocktype.externalvideo.php.utf8

TIA

[Matteo (matteo-scaramuccia)]

For the record:

# find /path/to/mahara-data/langpacks/it.utf8/ -name "*.php*" -print -type f -exec iconv -f UTF-8 -o --verbose /dev/null {} \;
[cut]
/path/to/mahara-data/langpacks/it.utf8/notification/internal/lang/it.utf8/notification.internal.php
iconv: illegal input sequence at position 1105
[cut]
/path/to/mahara-data/langpacks/it.utf8/blocktype/externalvideo/lang/it.utf8/blocktype.externalvideo.php.orig
iconv: illegal input sequence at position 1328
[cut]

so even notification.internal.php has the same typical issue with latin L10N, vowels with acute/grave accent: here "à".

[Matteo (matteo-scaramuccia)]

Functional wish for my question 1: in case of any failure find a reasonable recovery path, e.g. for L10N issues, fails over the English string. For my understanding after a quick view of the code this could not be an easy option ;) at least at json_reply level, maybe it could be done at get_string() level adding some extra checks into get_string_location(), not useful in a stable environment.
Since I'm really new to Mahara I'm not in the position to provide a good suggestion for other reasonable recovery paths for any other situation: the final need is that the user must be able to use Mahara even after such (rare?) events with no freeze at all.

[Matteo (matteo-scaramuccia)]

Patch proposal to mitigate (=no "freeze" of the UI) the issue. Maybe a starting point to add an error message instead of the (missing) configuration pane.

js/views.js:
--- C:\path\to\mahara-1.2.2/htdocs/js/views.js.orig Tue Dec 8 04:52:58 2009
+++ C:\path\to\mahara-1.2.2/htdocs/js/views.js Mon Feb 1 17:16:48 2010
@@ -338,6 +338,10 @@

     this.addConfigureBlock = function(oldblock, configblock, removeoncancel) {
+ if (!configblock.html) {
+ return;
+ }
+
         self.hideMediaPlayers();

         var temp = DIV();

[Richard Mansfield (richard-mansfield)]

I'm attaching a patch that will fix this problem at the get_string level by checking utf8-ness of every string and falling back to English for any that fail. I'm not sure if I'll apply it to core though -- it slows get_string down, and I think the correct approach is to get the language pack fixed.

[François Marier (fmarier)]

I agree that it should be fixed at the langpack level.

One idea would be to perform some basic sanity checks (e.g. "php -l file.php" and then an iconv check for utf8 correctness) for language packs as people commit to them. This could be added to our Hudson instance on ci.mahara.org

[Matteo (matteo-scaramuccia)]

See bug bug #561378: please take care of something like what proposed in comment 7, maybe also adding an hard-coded English message like:

  alert("Found a L10N issue with the current language pack!\r\nPlease contact your System Administrator.")

It's something useful regardless the specific issue found into the ITA language pack: it will give chances to recover issues when the object returned by the server through JSON will be corrupted someway.

[iwkse (iwkse)]

Changes related to this bug are present in last git version. Please download it.
If the bug still persists, feel free to reopen it.