maria.optimize corrupts stack around alloca() call

[Vladislav Vaintroub]

5.1, debug compilation, VS2010.
maria.optimize fails due to runtime check by compiler

maria.optimize [ fail ] Found warnings/errors in server log file!
        Test ended at 2010-12-06 14:09:08
line
Error:Run-Time Check Failure #4 - Stack area around _alloca memory reserved by this function is corrupted

[Vladislav Vaintroub (wlad-montyprogram)]

This is runtime check due to compiler option /RTC1.
The callstack leading to it is can be examined if mysqld.exe is started under debugger

perl mysql-test-run.pl --suite=maria optimize --debugger=devenv

> mysqld.exe!sort_one_index(st_handler_check_param * param, st_maria_handler * info, st_maria_keydef * keyinfo, unsigned __int64 pagepos, int new_file) Line 3211 + 0x15 bytes C
  mysqld.exe!sort_one_index(st_handler_check_param * param, st_maria_handler * info, st_maria_keydef * keyinfo, unsigned __int64 pagepos, int new_file) Line 3163 + 0x1d bytes C
  mysqld.exe!maria_sort_index(st_handler_check_param * param, st_maria_handler * info, char * name) Line 3036 + 0x2d bytes C
  mysqld.exe!ha_maria::repair(THD * thd, st_handler_check_param * param, bool do_optimize) Line 1526 + 0x1a bytes C++
  mysqld.exe!ha_maria::optimize(THD * thd, st_ha_check_opt * check_opt) Line 1403 + 0x12 bytes C++
  mysqld.exe!handler::ha_optimize(THD * thd, st_ha_check_opt * check_opt) Line 3296 + 0x1a bytes C++
  mysqld.exe!mysql_admin_table(THD * thd, TABLE_LIST * tables, st_ha_check_opt * check_opt, const char * operator_name, thr_lock_type lock_type, bool open_for_modify, bool no_warnings_for_error, unsigned int extra_open_options, int (THD *, TABLE_LIST *, st_ha_check_opt *)* prepare_func, int (THD *, st_ha_check_opt *)* operator_func, int (THD *, TABLE_LIST *)* view_operator_func) Line 4819 + 0x19 bytes C++
  mysqld.exe!mysql_optimize_table(THD * thd, TABLE_LIST * tables, st_ha_check_opt * check_opt) Line 5093 + 0x51 bytes C++
  mysqld.exe!mysql_execute_command(THD * thd) Line 3130 + 0x3d bytes C++
  mysqld.exe!mysql_parse(THD * thd, char * rawbuf, unsigned int length, const char * * found_semicolon) Line 6151 + 0x9 bytes C++
  mysqld.exe!dispatch_command(enum_server_command command, THD * thd, char * packet, unsigned int packet_length) Line 1273 + 0x22 bytes C++
  mysqld.exe!do_command(THD * thd) Line 899 + 0x1b bytes C++

[Vladislav Vaintroub (wlad-montyprogram)]

It looks like
 next_page= _ma_kpos(nod_flag,keypos);
statement in ma_check.c : sort_one_index()

overwrites at least one byte after the end of buff (that was allocated with my_alloca).

[Oleksandr "Sanja" Byelkin (sanja-byelkin)]

The bug is visible under valgrind if change my_alloca call with malloc() (not alloca())

[Michael Widenius (monty)]

Fix committed into 5.1