crash/segfault -#0 0x000000000082d7d7 in drizzled::TableShare::getTableTypeAsString ( this=0x2abd530) at ./drizzled/table_share.h:410 with multiple drizzledump clients

[Patrick Crews]

Crash in the server in an environment with 5 threads working on creating / altering tables and executing drizzledump. Appears to have happened during a drizzledump operation.

Not sure how realistic / likely this scenario is, but reporting it anyway.

Suspect this is best repeated via the random query generator.

randgen command line (assuming a server running @ port 9306):
./gentest.pl --dsn=dbi:drizzle:host=localhost:port=9306:user=root:password=:database=test --threads=5 --queries=1000 --engine=Innodb --gendata=conf/drizzle/drizzle.zz --grammar=conf/drizzle/drizzledump.yy --Validator=Drizzledump --queries=50

You can start drizzle with
./test-run --start-and-exit

Backtrace:
Program terminated with signal 11, Segmentation fault.
#0 0x000000000082d7d7 in drizzled::TableShare::getTableTypeAsString (
    this=0x2abd530) at ./drizzled/table_share.h:410
410 switch (table_proto->type())
#0 0x000000000082d7d7 in drizzled::TableShare::getTableTypeAsString (
    this=0x2abd530) at ./drizzled/table_share.h:410
#1 ShowTableStatus::Generator::fill (this=0x2abd530)
    at plugin/show_dictionary/show_table_status.cc:167
#2 0x000000000082da30 in ShowTableStatus::Generator::populate (
    this=0x2abd530) at plugin/show_dictionary/show_table_status.cc:142
#3 0x00000000005ebf11 in drizzled::plugin::TableFunction::Generator::sub_populate (this=0x2a9b7f8, field_size=12449368)
    at drizzled/plugin/table_function.cc:156
#4 0x00000000006d5dae in FunctionCursor::rnd_next (this=0x296f7d0)
    at plugin/function_engine/cursor.cc:87
#5 0x00000000005f01c1 in rr_sequential (info=0x2b4c040)
    at drizzled/records.cc:273
#6 0x0000000000628a71 in drizzled::sub_select (join=0x2ab2958,
    join_tab=0xbdf658, end_of_records=154) at drizzled/sql_select.cc:3565
#7 0x000000000062da70 in drizzled::do_select (join=0x2ab2958,
    fields=0x2aa93e0, table=0x0) at drizzled/sql_select.cc:3335
#8 0x00000000005a424c in drizzled::Join::exec (this=0x2ab2958)
    at drizzled/join.cc:1695
#9 0x0000000000629de2 in drizzled::mysql_select (session=0x2aa8680,
    rref_pointer_array=<value optimized out>, tables=0x2ab26a0, wild_num=1,
    fields=<value optimized out>, conds=<value optimized out>, og_num=0,
    order=0x0, group=0x0, having=0x0, select_options=2147500032,
    result=0x2ab2938, unit=0x2aa90e8, select_lex=0x2aa92e8)
    at drizzled/sql_select.cc:427
#10 0x000000000062a6e0 in drizzled::handle_select (session=0x2aa8680,
    lex=0x2aa90c8, result=0x2ab2938, setup_tables_done_option=0)
    at drizzled/sql_select.cc:146
#11 0x00000000006245e5 in drizzled::execute_sqlcom_select (session=0x2aa8680,
    all_tables=0x2ab26a0) at drizzled/sql_parse.cc:543
#12 0x000000000062422e in mysql_execute_command (session=0x2aa8680,
    inBuf=<value optimized out>, length=<value optimized out>)
    at drizzled/sql_parse.cc:478
#13 drizzled::mysql_parse (session=0x2aa8680, inBuf=<value optimized out>,
    length=<value optimized out>) at drizzled/sql_parse.cc:737
#14 0x00000000006265c2 in drizzled::dispatch_command (
    command=drizzled::COM_QUERY, session=0x2aa8680, packet=0x2a89911 "",
    packet_length=<value optimized out>) at drizzled/sql_parse.cc:223
#15 0x00000000005f367f in drizzled::Session::executeStatement (this=0x2aa8680)
    at drizzled/session.cc:659
#16 0x00000000005f5e82 in drizzled::Session::run (this=0x2aa8680)
    at drizzled/session.cc:518
#17 0x00007f35bcb852c4 in MultiThreadScheduler::runSession (
    arg=<value optimized out>) at ./plugin/multi_thread/multi_thread.h:67
#18 session_thread (arg=<value optimized out>)
    at plugin/multi_thread/multi_thread.cc:47
#19 0x00007f35bff309ca in start_thread (arg=<value optimized out>)
    at pthread_create.c:300
#20 0x00007f35bfc8d6fd in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#21 0x0000000000000000 in ?? ()

[Patrick Crews (patrick-crews)]

This does make me suspect that I want to stress out the database commands that drizzledump uses : )

[Patrick Crews (patrick-crews)]

Another, different backtrace. Different data (randgen run with --seed=time and --queries=25).
specific randgen seed = 1282080740

Documenting here for future research - this might be a different bug, but noting similar triggers.

Backtrace
Program terminated with signal 11, Segmentation fault.
#0 0x000000000082f67c in drizzled::Table::operator< (
    this=<value optimized out>, arg=<value optimized out>)
    at ./drizzled/table.h:613
613 if (this->getShare()->getTableProto()->type() < right.getShare()->getTableProto()->type())
#0 0x000000000082f67c in drizzled::Table::operator< (
    this=<value optimized out>, arg=<value optimized out>)
    at ./drizzled/table.h:613
#1 drizzled::Table::compare (this=<value optimized out>,
    arg=<value optimized out>) at ./drizzled/table.h:621
#2 __unguarded_linear_insert<__gnu_cxx::__normal_iterator<drizzled::Table**, std::vector<drizzled::Table*, std::allocator<drizzled::Table*> > >, drizzled::Table*, bool (*)(drizzled::Table const*, drizzled::Table const*)> (
    this=<value optimized out>, arg=<value optimized out>)
    at /usr/include/c++/4.4/bits/stl_algo.h:2084
#3 __insertion_sort<__gnu_cxx::__normal_iterator<drizzled::Table**, std::vector<drizzled::Table*, std::allocator<drizzled::Table*> > >, bool (*)(drizzled::Table const*, drizzled::Table const*)> (this=<value optimized out>,
    arg=<value optimized out>) at /usr/include/c++/4.4/bits/stl_algo.h:2134
#4 __final_insertion_sort<__gnu_cxx::__normal_iterator<drizzled::Table**, std::vector<drizzled::Table*, std::allocator<drizzled::Table*> > >, bool (*)(drizzled::Table const*, drizzled::Table const*)> (this=<value optimized out>,
    arg=<value optimized out>) at /usr/include/c++/4.4/bits/stl_algo.h:2193
#5 sort<__gnu_cxx::__normal_iterator<drizzled::Table**, std::vector<drizzled::Table*, std::allocator<drizzled::Table*> > >, bool (*)(drizzled::Table const*, drizzled::Table const*)> (this=<value optimized out>,
    arg=<value optimized out>) at /usr/include/c++/4.4/bits/stl_algo.h:5260
#6 Generator (this=<value optimized out>, arg=<value optimized out>)
    at plugin/show_dictionary/show_table_status.cc:78
#7 0x000000000082fd79 in ShowTableStatus::generator(drizzled::Field**) ()
#8 0x00000000006d5874 in FunctionCursor::doStartTableScan (this=0x355e040)
    at plugin/function_engine/cursor.cc:70
#9 0x0000000000503158 in drizzled::Cursor::startTableScan (this=0x355e040,
    scan=11) at drizzled/cursor.cc:150
#10 0x00000000005f090e in drizzled::ReadRecord::init_read_record (
    this=0x7f1dac01da10, session_arg=<value optimized out>,
    table_arg=<value optimized out>, select_arg=0x7f1dac01dd60,
    use_record_cache=<value optimized out>, print_error_arg=48)
    at drizzled/records.cc:158
#11 0x00000000006288e8 in join_init_read_record (tab=0x7f1dac01d988)
    at drizzled/sql_select.cc:3969
#12 0x0000000000628a71 in drizzled::sub_select (join=0x35cbd88,
    join_tab=0x7f1da409f60b, end_of_records=false)
    at drizzled/sql_select.cc:3565
#13 0x000000000062da70 in drizzled::do_select (join=0x35cbd88,
    fields=0x3609f00, table=0x0) at drizzled/sql_select.cc:3335
#14 0x00000000005a424c in drizzled::Join::exec (this=0x35cbd88)
  ...

[PrafullaT (prafulla-t)]

Hi Patrick,
I am trying to look into this bug.
But I am not able to reproduce it.
Here is what I get when I run genTest.pl from command line.
Could you please let me know If I am doing anything wrong ?
I see this error on gen_test output
This looks to be wrong.

# 2010-08-20T09:52:46 Started periodic reporting process...
# 2010-08-20T09:52:46 Query: USE drizzledump_db failed: 1049 Unknown database '%s'. Further errors of this kind will be suppressed.

I have attached entire ./genTest.pl output for your ref

[Patrick Crews (patrick-crews)]

Hi Prafulla,

The crashes we saw might have been a fortunate mistake of timing - always a
problem with multi-threaded randomness : (

I'll give it a few runs on my box and update the bug with my findings / any
info to help reproduce this more reliably.

Thanks for checking this out.

On Fri, Aug 20, 2010 at 12:24 AM, Prafulla Tekawade <
<email address hidden>> wrote:

> Hi Patrick,
> I am trying to look into this bug.
> But I am not able to reproduce it.
> Here is what I get when I run genTest.pl from command line.
> Could you please let me know If I am doing anything wrong ?
> I see this error on gen_test output
> This looks to be wrong.
>
> # 2010-08-20T09:52:46 Started periodic reporting process...
> # 2010-08-20T09:52:46 Query: USE drizzledump_db failed: 1049 Unknown
> database '%s'. Further errors of this kind will be suppressed.
>
> I have attached entire ./genTest.pl output for your ref
>
> ** Attachment added: "genTest.pl output"
>
> https://bugs.launchpad.net/drizzle/+bug/619492/+attachment/1502410/+files/bug_out_expr
>
> --
> crash/segfault -#0 0x000000000082d7d7 in
> drizzled::TableShare::getTableTypeAsString ( this=0x2abd530) at
> ./drizzled/table_share.h:410 with multiple drizzledump clients
> https://bugs.launchpad.net/bugs/619492
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in A Lightweight SQL Database for Cloud and Web: Confirmed
>
> Bug description:
> Crash in the server in an environment with 5 threads working on creating /
> altering tables and executing drizzledump. Appears to have happened during
> a drizzledump operation.
>
> Not sure how realistic / likely this scenario is, but reporting it anyway.
>
> Suspect this is best repeated via the random query generator.
>
> randgen command line (assuming a server running @ port 9306):
> ./gentest.pl--dsn=dbi:drizzle:host=localhost:port=9306:user=root:password=:database=test
> --threads=5 --queries=1000 --engine=Innodb --gendata=conf/drizzle/drizzle.zz
> --grammar=conf/drizzle/drizzledump.yy --Validator=Drizzledump --queries=50
>
> You can start drizzle with
> ./test-run --start-and-exit
>
> Backtrace:
> Program terminated with signal 11, Segmentation fault.
> #0 0x000000000082d7d7 in drizzled::TableShare::getTableTypeAsString (
> this=0x2abd530) at ./drizzled/table_share.h:410
> 410 switch (table_proto->type())
> #0 0x000000000082d7d7 in drizzled::TableShare::getTableTypeAsString (
> this=0x2abd530) at ./drizzled/table_share.h:410
> #1 ShowTableStatus::Generator::fill (this=0x2abd530)
> at plugin/show_dictionary/show_table_status.cc:167
> #2 0x000000000082da30 in ShowTableStatus::Generator::populate (
> this=0x2abd530) at plugin/show_dictionary/show_table_status.cc:142
> #3 0x00000000005ebf11 in
> drizzled::plugin::TableFunction::Generator::sub_populate (this=0x2a9b7f8,
> field_size=12449368)
> at drizzled/plugin/table_function.cc:156
> #4 0x00000000006d5dae in FunctionCursor::rnd_next (this=0x296f7d0)
> at plugin/function_engine/cursor.cc:87
> #5 0x00000000005f01c1 in rr_sequential (info=0x2b4c040)
> at drizzled/records.cc:273
> #6 0x0000000000628a71 in drizzled::sub_select (...

[Patrick Crews (patrick-crews)]

Prafulla,

I was able to crash my server pretty easily with the randgen command line shown above.

You will some of this output you mention as we'll have 5 threads competing to do the exact same thing - some collisions are expected.

It might be you need to up the value of --queries in order to give your box more of a chance to hit a juicy, crashing collision : )
./gentest.pl --dsn=dbi:drizzle:host=localhost:port=9306:user=root:password=:database=test --threads=5 --queries=1000 --engine=Innodb --gendata=conf/drizzle/drizzle.zz --grammar=conf/drizzle/drizzledump.yy --Validator=Drizzledump --queries=50
--debug.

You could probably go up to --queries=100 or even 200 if you aren't seeing it at 50. You could also add in --seed=time to randomize the queries generated + data to see about hitting something.

I've added the debug switch to the command line as it will help indicate whether or not Drizzle has crashed. The randgen isn't currently as informative as it should be with Drizzle (works far better on MySQL). As a result, we have to manually check for crashes : (

What debug does is print out each query executed and collects various randgen statistics during the run. At the end of the run, you'll see output as noted here:
http://drizzle.org/wiki/Randgen_running_tests#--debug_is_your_friend

If you have debug on and you *don't* see this output, it is indicative that you have a crash. You can then call gdb against the server and see what kind of havok you have managed to wreak on the server : )

Thanks for the help. I hope this information helps and just let me know if I can do anything else.