strconvert() / my_mb_wc_filename() can do out-of-bound reads on short inputs

[Alexey Kopytov]

Found using AddressSanitizer testing.

strconvert() makes the following assumption about the character set's 'mb_wc' method:

    /*
      Using 'from + 10' is safe:
      - it is enough to scan a single character in any character set.
      - if remaining string is shorter than 10, then mb_wc will return
        with error because of unexpected '\0' character.
    */

However, my_mb_wc_filename() (perhaps other implementations as well, I didn't check other handlers) does not stop immediately when encountering the '\0' character. If the first character of the input string is not '\0', it assumes the input to contain at least 2 more characters:

  if (s + 3 > e)
    return MY_CS_TOOSMALL3;

  byte1= s[1];
  byte2= s[2];

Which may result in an out-of-bound read of at most 2 bytes and thus, undefined program behavior.

In particular, an out-of-bound read of 1 byte occurs during InnoDB initialization. innodb_init() calls filename_to_tablename() with a 2-char 'from' argument string, so the above assumptions in strconvert() / my_mb_wc_filename() do not hold.

How to repeat:
Analyze the code path in innodb_init() -> filename_to_tablename() -> strconvert() -> my_mb_wc_filename(), or start an AddressSanitizer-instrumented mysqld with InnoDB enabled.

[Laurynas Biveinis (laurynas-biveinis)]

Likely fixed under internal bug id 14834378 in 5.5.33 / 5.6.9.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-596