intermittent ER_KEY_NOT_FOUND errors during insert

[Mark Callaghan]

I am using MySQL 5.1.45 and PBXT 1.1 revision 851

my non-default pbxt settings are:
pbxt_index_cache_size=250M
pbxt_record_cache_size=750M
pbxt_flush_log_at_trx_commit=2

I am using a modified sysbench test (to be published real soon) in which 1 or more threads run nothing but the following on the sbtest table:
"INSERT INTO %s values(?,0,'0',"
           "'aaaaaaaaaaffffffffffrrrrrrrrrreeeeeeeeeeyyyyyyyyyy')"
           " ON DUPLICATE KEY UPDATE c = c + 1",

C has type 'char' in the table, I use it as an int.

I get intermittent errors:
ALERT: failed to execute MySQL query: `INSERT INTO sbtest values(1,0,'0','aaaaaaaaaaffffffffffrrrrrrrrrreeeeeeeeeey
yyyyyyyyy') ON DUPLICATE KEY UPDATE c = c + 1`:
ALERT: Error 1032 Can't find record in 'sbtest'
FATAL: database error, exiting...

I run it in a loop: first for 1 thread, then 2, then 4, ... then 1024
The test was OK for 1 thread.
Then failed for 2, 4, 8, 16, 32 threads
Then was ok for 64 or more threads

There are no errors in the database error log.

[Mark Callaghan (mdcallag)]

Reproduction code is sysbench at https://code.launchpad.net/~mdcallag/sysbench/0.4-incupdate
vary the the value for --num-threads

sysbench --test=oltp --oltp-table-size=2000000 --max-time=60 --max-requests=0 --mysql-table-engine=pbxt \
  --db-ps-mode=disable --mysql-engine-trx=no --oltp-skip-trx --oltp-test-mode=incinsert \
  --oltp-dist-type=uniform --oltp-range-size=1 prepare

sysbench --test=oltp --oltp-table-size=2000000 --max-time=60 --max-requests=0 --mysql-table-engine=pbxt \
--db-ps-mode=disable --mysql-engine-trx=no --oltp-skip-trx --oltp-test-mode=incinsert \
--oltp-dist-type=uniform --oltp-range-size=1 --num-threads=1 --seed-rng=1 run

[Paul McCullagh (paul-mccullagh)]

Hi Mark,

Thanks for the bug report. We will check it out...

[Paul McCullagh (paul-mccullagh)]

I cannnot repeat this bug with PBXT 1.0.11. Most likely due to this bug fix:

RN318: Fixed a bug in the atomic R/W lock. This bug occurred on multi-core Linux when under extrem load. The affect was that an index lookup could fail. The index was not corrupted.

[Paul McCullagh (paul-mccullagh)]

I believe this bug is fixed in 1.0.11, which has already been pushed to 1.1, most like by this change in RN318.

Mark, if you can, please confirm the fix, and re-open the bug if this is not the case.

[Mark Callaghan (mdcallag)]

great, eventually I will do that. Our devel build now includes pbxt.

On Thu, May 27, 2010 at 9:00 AM, Paul McCullagh <
<email address hidden>> wrote:

> I believe this bug is fixed in 1.0.11, which has already been pushed to
> 1.1, most like by this change in RN318.
>
> Mark, if you can, please confirm the fix, and re-open the bug if this is
> not the case.
>
>
> ** Changed in: pbxt
> Status: In Progress => Fix Committed
>
> --
> intermittent ER_KEY_NOT_FOUND errors during insert
> https://bugs.launchpad.net/bugs/551250
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in PrimeBase XT: Fix Committed
>
> Bug description:
> I am using MySQL 5.1.45 and PBXT 1.1 revision 851
>
> my non-default pbxt settings are:
> pbxt_index_cache_size=250M
> pbxt_record_cache_size=750M
> pbxt_flush_log_at_trx_commit=2
>
> I am using a modified sysbench test (to be published real soon) in which 1
> or more threads run nothing but the following on the sbtest table:
> "INSERT INTO %s values(?,0,'0',"
> "'aaaaaaaaaaffffffffffrrrrrrrrrreeeeeeeeeeyyyyyyyyyy')"
> " ON DUPLICATE KEY UPDATE c = c + 1",
>
> C has type 'char' in the table, I use it as an int.
>
> I get intermittent errors:
> ALERT: failed to execute MySQL query: `INSERT INTO sbtest
> values(1,0,'0','aaaaaaaaaaffffffffffrrrrrrrrrreeeeeeeeeey
> yyyyyyyyy') ON DUPLICATE KEY UPDATE c = c + 1`:
> ALERT: Error 1032 Can't find record in 'sbtest'
> FATAL: database error, exiting...
>
> I run it in a loop: first for 1 thread, then 2, then 4, ... then 1024
> The test was OK for 1 thread.
> Then failed for 2, 4, 8, 16, 32 threads
> Then was ok for 64 or more threads
>
> There are no errors in the database error log.
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/pbxt/+bug/551250/+subscribe
>

--
Mark Callaghan
<email address hidden>