Bug #607946 “Crash / assertion failed in drizzled::Cursor::ha_re...” : Bugs : Drizzle

Revision history for this message

Patrick Crews (patrick-crews) wrote on 2010-07-20:

#1

So far, I have been unable to repeat this on my machine with the same command line. Even increasing the number of queries and threads hasn't worked. This could have been a quirk of timing on the test machine. Will re-run and see if it is repeatable there once initial randgen run is completed.

Revision history for this message

Patrick Crews (patrick-crews) wrote on 2010-07-20:

#2

Passed on re-test on Hudson machine. Seems like a hard to hit bug as I've tried several iterations of standard runs and randomized query / test data runs without a repeat.

Changed in drizzle:
status:	New → Opinion

Revision history for this message

Patrick Crews (patrick-crews) wrote on 2010-08-11:

#3

UPDATE:
This crash hasn't been seen since it's initial detection on hudson.

Revision history for this message

Jason Belich (jason-belich) wrote on 2010-10-01:

#4

I'm getting this same error by slamming my local drizzle (2010.09.1802-1ubuntu1~lucid1 debian package) instance with a couple million of this query (in php using PDO_Mysql) :

INSERT INTO audit_count (database_name, table_name, column_name, html_count, xss_count ) VALUES (?,?,?,?,?) ON DUPLICATE KEY UPDATE html_count = html_count + ?, xss_count = xss_count + ?

And the table structure is this:

CREATE TABLE IF NOT EXISTS `audit_count` (
`database_name` CHAR(64),
`table_name` CHAR(64),
`column_name` CHAR(64),
`html_count` integer,
`xss_count` integer,
PRIMARY KEY (
  `database_name`,
  `table_name`,
  `column_name`
)
);

And here is the error i'm getting out of the stderr:

drizzled: drizzled/cursor.cc:1460: int drizzled::Cursor::ha_reset(): Assertion `table->getShare()->all_set.isSetAll()' failed.
101001 10:42:51 - drizzled got signal 6;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

read_buffer_size=131072
max_used_connections=6
connection_count=5
It is possible that drizzled could use up to
(read_buffer_size + sort_buffer_size)*thread_count
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Number of stack frames obtained: 15
()
gsignal()
abort()
__assert_fail()
drizzled::Cursor::ha_reset()
drizzled::Session::free_cached_table()
drizzled::Session::close_open_tables()
drizzled::dispatch_command(drizzled::enum_server_command, drizzled::Session*, char*, unsigned int)
drizzled::Session::executeStatement()
drizzled::Session::run()
MultiThreadScheduler::runSession(drizzled::Session*)
thread_proxy()
()
clone()

You can guess what i'm doing here by the query and schema. I'm running 5 php gearman workers which read from gearman and pump the numbers into the table. super simple... It seems to be most unstable under heavy loads.

I'm getting this same error by slamming my local drizzle (2010.09.1802-1ubuntu1~lucid1 debian package) instance with a couple million of this query (in php using PDO_Mysql) :

INSERT INTO audit_count (database_name, table_name, column_name, html_count, xss_count ) VALUES (?,?,?,?,?) ON DUPLICATE KEY UPDATE html_count = html_count + ?, xss_count = xss_count + ?

And the table structure is this:

CREATE TABLE IF NOT EXISTS `audit_count` (
	`database_name` CHAR(64),
	`table_name` CHAR(64),
	`column_name` CHAR(64),
	`html_count` integer,
	`xss_count` integer,
	PRIMARY KEY (
		`database_name`,
		`table_name`,
		`column_name`
	)
);

And here is the error i'm getting out of the stderr:

drizzled: drizzled/cursor.cc:1460: int drizzled::Cursor::ha_reset(): Assertion `table->getShare()->all_set.isSetAll()' failed.
101001 10:42:51 - drizzled got signal 6;
This could be because you hit a bug. It is also possible that this binary
 or one of the libraries it was linked against is corrupt, improperly built,
 or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

read_buffer_size=131072
max_used_connections=6
connection_count=5
It is possible that drizzled could use up to 
(read_buffer_size + sort_buffer_size)*thread_count
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Number of stack frames obtained: 15
()
gsignal()
abort()
__assert_fail()
drizzled::Cursor::ha_reset()
drizzled::Session::free_cached_table()
drizzled::Session::close_open_tables()
drizzled::dispatch_command(drizzled::enum_server_command, drizzled::Session*, char*, unsigned int)
drizzled::Session::executeStatement()
drizzled::Session::run()
MultiThreadScheduler::runSession(drizzled::Session*)
thread_proxy()
()
clone()

You can guess what i'm doing here by the query and schema.  I'm running 5 php gearman workers which read from gearman and pump the numbers into the table.  super simple... It seems to be most unstable under heavy loads.

Revision history for this message

Jason Belich (jason-belich) wrote on 2010-10-01:

#5

and oh yes.. I'm starting drizzle with the simple

drizzled --no-defaults --datadir=/opt/drizzle/ >> /var/log/drizzle/drizzled.log 2>&1 &

Changed in drizzle:
status:	Opinion → New

Revision history for this message

Monty Taylor (mordred) wrote on 2010-10-01: Re: [Bug 607946] Re: Crash / assertion failed in drizzled::Cursor::ha_reset (this=0x7ffbbc004ac8) at drizzled/cursor.cc:1455

#6

EXCELLENT. thanks for the bug ... looking forward to figuring out what's
going on there!

On 10/01/2010 01:02 PM, Jason Belich wrote:
> I'm getting this same error by slamming my local drizzle
> (2010.09.1802-1ubuntu1~lucid1 debian package) instance with a couple
> million of this query (in php using PDO_Mysql) :
>
> INSERT INTO audit_count (database_name, table_name, column_name,
> html_count, xss_count ) VALUES (?,?,?,?,?) ON DUPLICATE KEY UPDATE
> html_count = html_count + ?, xss_count = xss_count + ?
>
> And the table structure is this:
>
> CREATE TABLE IF NOT EXISTS `audit_count` (
> `database_name` CHAR(64),
> `table_name` CHAR(64),
> `column_name` CHAR(64),
> `html_count` integer,
> `xss_count` integer,
> PRIMARY KEY (
> `database_name`,
> `table_name`,
> `column_name`
> )
> );
>
> And here is the error i'm getting out of the stderr:
>
> drizzled: drizzled/cursor.cc:1460: int drizzled::Cursor::ha_reset(): Assertion `table->getShare()->all_set.isSetAll()' failed.
> 101001 10:42:51 - drizzled got signal 6;
> This could be because you hit a bug. It is also possible that this binary
> or one of the libraries it was linked against is corrupt, improperly built,
> or misconfigured. This error can also be caused by malfunctioning hardware.
> We will try our best to scrape up some info that will hopefully help diagnose
> the problem, but since we have already crashed, something is definitely wrong
> and this may fail.
>
> read_buffer_size=131072
> max_used_connections=6
> connection_count=5
> It is possible that drizzled could use up to
> (read_buffer_size + sort_buffer_size)*thread_count
> bytes of memory
> Hope that's ok; if not, decrease some variables in the equation.
>
> Number of stack frames obtained: 15
> ()
> gsignal()
> abort()
> __assert_fail()
> drizzled::Cursor::ha_reset()
> drizzled::Session::free_cached_table()
> drizzled::Session::close_open_tables()
> drizzled::dispatch_command(drizzled::enum_server_command, drizzled::Session*, char*, unsigned int)
> drizzled::Session::executeStatement()
> drizzled::Session::run()
> MultiThreadScheduler::runSession(drizzled::Session*)
> thread_proxy()
> ()
> clone()
>
>
> You can guess what i'm doing here by the query and schema. I'm running 5 php gearman workers which read from gearman and pump the numbers into the table. super simple... It seems to be most unstable under heavy loads.
>

EXCELLENT. thanks for the bug ... looking forward to figuring out what's
going on there!

On 10/01/2010 01:02 PM, Jason Belich wrote:
> I'm getting this same error by slamming my local drizzle
> (2010.09.1802-1ubuntu1~lucid1 debian package) instance with a couple
> million of this query (in php using PDO_Mysql) :
> 
> INSERT INTO audit_count (database_name, table_name, column_name,
> html_count, xss_count ) VALUES (?,?,?,?,?) ON DUPLICATE KEY UPDATE
> html_count = html_count + ?, xss_count = xss_count + ?
> 
> And the table structure is this:
> 
> CREATE TABLE IF NOT EXISTS `audit_count` (
> 	`database_name` CHAR(64),
> 	`table_name` CHAR(64),
> 	`column_name` CHAR(64),
> 	`html_count` integer,
> 	`xss_count` integer,
> 	PRIMARY KEY (
> 		`database_name`,
> 		`table_name`,
> 		`column_name`
> 	)
> );
> 
> And here is the error i'm getting out of the stderr:
> 
> drizzled: drizzled/cursor.cc:1460: int drizzled::Cursor::ha_reset(): Assertion `table->getShare()->all_set.isSetAll()' failed.
> 101001 10:42:51 - drizzled got signal 6;
> This could be because you hit a bug. It is also possible that this binary
>  or one of the libraries it was linked against is corrupt, improperly built,
>  or misconfigured. This error can also be caused by malfunctioning hardware.
> We will try our best to scrape up some info that will hopefully help diagnose
> the problem, but since we have already crashed, something is definitely wrong
> and this may fail.
> 
> read_buffer_size=131072
> max_used_connections=6
> connection_count=5
> It is possible that drizzled could use up to 
> (read_buffer_size + sort_buffer_size)*thread_count
> bytes of memory
> Hope that's ok; if not, decrease some variables in the equation.
> 
> Number of stack frames obtained: 15
> ()
> gsignal()
> abort()
> __assert_fail()
> drizzled::Cursor::ha_reset()
> drizzled::Session::free_cached_table()
> drizzled::Session::close_open_tables()
> drizzled::dispatch_command(drizzled::enum_server_command, drizzled::Session*, char*, unsigned int)
> drizzled::Session::executeStatement()
> drizzled::Session::run()
> MultiThreadScheduler::runSession(drizzled::Session*)
> thread_proxy()
> ()
> clone()
> 
> 
> You can guess what i'm doing here by the query and schema.  I'm running 5 php gearman workers which read from gearman and pump the numbers into the table.  super simple... It seems to be most unstable under heavy loads.
>

Revision history for this message

Patrick Crews (patrick-crews) wrote on 2010-10-01: Re: [Bug 607946] Re: Crash / assertion failed in drizzled::Cursor::ha_reset (this=0x7ffbbc004ac8) at drizzled/cursor.cc:1455

#7

Download full text (5.7 KiB)

Thanks for this input! Awesome to know someone else can see it so we can
get to work on a fix : )

On Fri, Oct 1, 2010 at 2:34 PM, Monty Taylor <email address hidden> wrote:

> EXCELLENT. thanks for the bug ... looking forward to figuring out what's
> going on there!
>
> On 10/01/2010 01:02 PM, Jason Belich wrote:
> > I'm getting this same error by slamming my local drizzle
> > (2010.09.1802-1ubuntu1~lucid1 debian package) instance with a couple
> > million of this query (in php using PDO_Mysql) :
> >
> > INSERT INTO audit_count (database_name, table_name, column_name,
> > html_count, xss_count ) VALUES (?,?,?,?,?) ON DUPLICATE KEY UPDATE
> > html_count = html_count + ?, xss_count = xss_count + ?
> >
> > And the table structure is this:
> >
> > CREATE TABLE IF NOT EXISTS `audit_count` (
> > `database_name` CHAR(64),
> > `table_name` CHAR(64),
> > `column_name` CHAR(64),
> > `html_count` integer,
> > `xss_count` integer,
> > PRIMARY KEY (
> > `database_name`,
> > `table_name`,
> > `column_name`
> > )
> > );
> >
> > And here is the error i'm getting out of the stderr:
> >
> > drizzled: drizzled/cursor.cc:1460: int drizzled::Cursor::ha_reset():
> Assertion `table->getShare()->all_set.isSetAll()' failed.
> > 101001 10:42:51 - drizzled got signal 6;
> > This could be because you hit a bug. It is also possible that this binary
> > or one of the libraries it was linked against is corrupt, improperly
> built,
> > or misconfigured. This error can also be caused by malfunctioning
> hardware.
> > We will try our best to scrape up some info that will hopefully help
> diagnose
> > the problem, but since we have already crashed, something is definitely
> wrong
> > and this may fail.
> >
> > read_buffer_size=131072
> > max_used_connections=6
> > connection_count=5
> > It is possible that drizzled could use up to
> > (read_buffer_size + sort_buffer_size)*thread_count
> > bytes of memory
> > Hope that's ok; if not, decrease some variables in the equation.
> >
> > Number of stack frames obtained: 15
> > ()
> > gsignal()
> > abort()
> > __assert_fail()
> > drizzled::Cursor::ha_reset()
> > drizzled::Session::free_cached_table()
> > drizzled::Session::close_open_tables()
> > drizzled::dispatch_command(drizzled::enum_server_command,
> drizzled::Session*, char*, unsigned int)
> > drizzled::Session::executeStatement()
> > drizzled::Session::run()
> > MultiThreadScheduler::runSession(drizzled::Session*)
> > thread_proxy()
> > ()
> > clone()
> >
> >
> > You can guess what i'm doing here by the query and schema. I'm running 5
> php gearman workers which read from gearman and pump the numbers into the
> table. super simple... It seems to be most unstable under heavy loads.
> >
>
> --
> Crash / assertion failed in drizzled::Cursor::ha_reset
> (this=0x7ffbbc004ac8) at drizzled/cursor.cc:1455
> https://bugs.launchpad.net/bugs/607946
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in A Lightweight SQL Database for Cloud Infrastructure and Web
> Applications: New
>
> Bug description:
> A new crash was detected in randgen testing:
> ht...

Thanks for this input!  Awesome to know someone else can see it so we can
get to work on a fix : )

On Fri, Oct 1, 2010 at 2:34 PM, Monty Taylor <mordred@inaugust.com> wrote:

> EXCELLENT. thanks for the bug ... looking forward to figuring out what's
> going on there!
>
> On 10/01/2010 01:02 PM, Jason Belich wrote:
> > I'm getting this same error by slamming my local drizzle
> > (2010.09.1802-1ubuntu1~lucid1 debian package) instance with a couple
> > million of this query (in php using PDO_Mysql) :
> >
> > INSERT INTO audit_count (database_name, table_name, column_name,
> > html_count, xss_count ) VALUES (?,?,?,?,?) ON DUPLICATE KEY UPDATE
> > html_count = html_count + ?, xss_count = xss_count + ?
> >
> > And the table structure is this:
> >
> > CREATE TABLE IF NOT EXISTS `audit_count` (
> >       `database_name` CHAR(64),
> >       `table_name` CHAR(64),
> >       `column_name` CHAR(64),
> >       `html_count` integer,
> >       `xss_count` integer,
> >       PRIMARY KEY (
> >               `database_name`,
> >               `table_name`,
> >               `column_name`
> >       )
> > );
> >
> > And here is the error i'm getting out of the stderr:
> >
> > drizzled: drizzled/cursor.cc:1460: int drizzled::Cursor::ha_reset():
> Assertion `table->getShare()->all_set.isSetAll()' failed.
> > 101001 10:42:51 - drizzled got signal 6;
> > This could be because you hit a bug. It is also possible that this binary
> >  or one of the libraries it was linked against is corrupt, improperly
> built,
> >  or misconfigured. This error can also be caused by malfunctioning
> hardware.
> > We will try our best to scrape up some info that will hopefully help
> diagnose
> > the problem, but since we have already crashed, something is definitely
> wrong
> > and this may fail.
> >
> > read_buffer_size=131072
> > max_used_connections=6
> > connection_count=5
> > It is possible that drizzled could use up to
> > (read_buffer_size + sort_buffer_size)*thread_count
> > bytes of memory
> > Hope that's ok; if not, decrease some variables in the equation.
> >
> > Number of stack frames obtained: 15
> > ()
> > gsignal()
> > abort()
> > __assert_fail()
> > drizzled::Cursor::ha_reset()
> > drizzled::Session::free_cached_table()
> > drizzled::Session::close_open_tables()
> > drizzled::dispatch_command(drizzled::enum_server_command,
> drizzled::Session*, char*, unsigned int)
> > drizzled::Session::executeStatement()
> > drizzled::Session::run()
> > MultiThreadScheduler::runSession(drizzled::Session*)
> > thread_proxy()
> > ()
> > clone()
> >
> >
> > You can guess what i'm doing here by the query and schema.  I'm running 5
> php gearman workers which read from gearman and pump the numbers into the
> table.  super simple... It seems to be most unstable under heavy loads.
> >
>
> --
> Crash / assertion failed in drizzled::Cursor::ha_reset
> (this=0x7ffbbc004ac8)     at drizzled/cursor.cc:1455
> https://bugs.launchpad.net/bugs/607946
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in A Lightweight SQL Database for Cloud Infrastructure and Web
> Applications: New
>
> Bug description:
> A new crash was detected in randgen testing:
> http://hudson.drizzle.org/job/drizzle-staging-randgen/417/
>
> This occurred with the following randgen command line:
> ./gentest.pl--dsn=dbi:drizzle:host=localhost:port=9306:user=root:password=:database=test
> --threads=5 --queries=1000 --engine=Innodb
> --gendata=conf/transactions/transactions.zz
> --grammar=conf/transactions/repeatable_read.yy --validators=RepeatableRead
>
> Still working to reproduce on command / produce a test case.  Unable to
> reproduce this on my machine at present.
>
> Backtrace:
> Program terminated with signal 6, Aborted.
> #0  0x00007ffbdecb14b5 in *__GI_raise (sig=<value optimized out>)
>    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> 64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
>        in ../nptl/sysdeps/unix/sysv/linux/raise.c
> Current language:  auto
> The current source language is "auto; currently c".
> #0  0x00007ffbdecb14b5 in *__GI_raise (sig=<value optimized out>)
>    at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1  0x00007ffbdecb4f50 in *__GI_abort () at abort.c:92
> #2  0x00007ffbdecaa481 in *__GI___assert_fail (
>    assertion=0x8251e0 "table->getShare()->all_set.isSetAll()",
>    file=<value optimized out>, line=1455,
>    function=0x825580 "int drizzled::Cursor::ha_reset()") at assert.c:81
> #3  0x00000000005012e2 in drizzled::Cursor::ha_reset (this=0x7ffbbc004ac8)
>    at drizzled/cursor.cc:1455
> #4  0x00000000006011bf in drizzled::Session::free_cached_table (
>    this=<value optimized out>) at drizzled/sql_base.cc:518
> #5  0x0000000000601288 in drizzled::Session::close_open_tables (
>    this=0x24403e0) at drizzled/sql_base.cc:546
> #6  0x000000000061cace in drizzled::dispatch_command (
>    command=drizzled::COM_QUERY, session=0x24403e0, packet=0x2430b91 "",
>    packet_length=0) at drizzled/sql_parse.cc:306
> #7  0x00000000005ed91f in drizzled::Session::executeStatement
> (this=0x24403e0)
>    at drizzled/session.cc:648
> #8  0x00000000005ef712 in drizzled::Session::run (this=0x24403e0)
>    at drizzled/session.cc:505
> #9  0x00007ffbdba6c024 in MultiThreadScheduler::runSession (
>    arg=<value optimized out>) at ./plugin/multi_thread/multi_thread.h:67
> #10 session_thread (arg=<value optimized out>)
>    at plugin/multi_thread/multi_thread.cc:43
> #11 0x00007ffbdeff3a04 in start_thread (arg=<value optimized out>)
>    at pthread_create.c:300
> #12 0x00007ffbded5d80d in clone ()
>    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
> #13 0x0000000000000000 in ?? ()
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/drizzle/+bug/607946/+subscribe
>

Revision history for this message

Andrew Hutchings (linuxjedi) wrote on 2010-12-17:

#8

ok, so I haven't been able to reproduce this using Jason's test and the assert() test (amongst other things) has been changed since the bug was opened.

Jason: are you still able to reproduce this?

Revision history for this message

Andrew Hutchings (linuxjedi) wrote on 2011-01-10:

#9

Marking as fix released, can no longer reproduce and unlikely to be able to hit the same bug again given changes to cursor

Affects		Status	Importance	Assigned to	Milestone
	Drizzle	Fix Released	Undecided	Andrew Hutchings	Drizzle 2011-01-17
	7.0	Fix Released	Medium	Andrew Hutchings	Drizzle 2011-01-17

Drizzle

Crash / assertion failed in drizzled::Cursor::ha_reset (this=0x7ffbbc004ac8) at drizzled/cursor.cc:1455

Bug Description

Other bug subscribers

Remote bug watches