'ERROR -86: Too many threads' reported during concurrecy test

[Paul McCullagh]

For a test which runs 50 client connections, neither default pbxt_max_threads=127 nor increased up to 256 was not enough.

The following error was reported multiple times:

Error] void thr_init(thread_xt.cc:1124) -86: Too many threads: 129, increase pbxt_max_threads

[Paul McCullagh (paul-mccullagh)]

This looks like a bug in PBXT, but I am not sure if it is so easy to repeat because we have done tests with large numbers of threads.

I also did a quick test with 128 insert threads using PBXT and 6.0.9 without generating any errors.

Please let us know exactly how the test was done.

[Paul McCullagh (paul-mccullagh)]

When this error occurs, the PBXT statistics also stop working. This may be a related problem.

[Elena Stepanova (elenst)]

In my test configuration, it seems to be a consequence of the problem described in bug#430641.

A real stress test in addition to the main test flow also sets up several events which are run with different intervals (1-10 minutes).
The test is run with 50 client connections. When it gets stuck at the point described in bug#430641, processes started by client connections lock some data; since the same data is supposed to be used by the events, new events cannot finish and pile up as running processes, Thus, even also the number of connected threads is fixed, the number of running threads keeps increasing and eventually hits the limit.

[Elena Stepanova (elenst)]

The initial description related to a test which was run with max_connections=120, set as a command-line parameter on server startup.
The same problem happens if a test is run with default max_connections value 151 (pbxt_max_threads=158).
In both cases neither max_connections nor pbxt_max_threads are changed during the test.
.
The attached archive contains a test which might help to reproduce the problem. It executes exactly the same flow as the test provided for bug#430641, but it also turns on event scheduler and sets up several events (with very short time intervals to get processes pile up faster).

The usage is exactly the same as for the test in bug#430641.

[Paul McCullagh (paul-mccullagh)]

This bug will be fixed in 1.1.

Currently the only solution is to set the variable pbxt_max_threads high enough that the internal thread requirement is not exceeded. pbxt_max_threads cannot be updated dynamically so the server must be restarted to change the value. Setting pbxt_max_threads to zero cause PBXT to estimate the thread requirement based on the max_connections setting. So depending on the number of background tasks run by MySQL, it is always possible to run into the error described here.

In 1.1 we plan to change the semantics of pbxt_max_threads. pbxt_max_threads will then be an upper limit to the number of threads that may be created. Set to zero will mean unlimited.

In order to do this, the thread array inside PBXT must be dynamically reallocated, or we must remove the dependency on this array. This is seen as a significant change which could affect stability and is therefore not desirable in the current RC series.

[Paul McCullagh (paul-mccullagh)]

Given the archive above, this is how to repeat the bug:

mkdir /home/build/max-thread-bug
cd /home/build/max-thread-bug

# Unpack the archive (it creates a directory called pbxt).

# Copy mysql-stress-test.pl to this directory.

# Remove these directories to re-run the test:
rm -r t ; rm -r r

./mysql-stress-test.pl --stress-basedir=/home/build/max-thread-bug --stress-suite-basedir=/home/build/max-thread-bug/pbxt --server-logs-dir=/home/build/max-thread-bug/pbxt/logs --stress-init-file=/home/build/max-thread-bug/pbxt/initdb_evnt_part.txt --stress-tests-file=/home/build/max-thread-bug/pbxt/iuds3_tests.txt --stress-datadir=/home/build/max-thread-bug

[Paul McCullagh (paul-mccullagh)]

I have removed the system variable pbxt_max_threads. This is now handled dynamically by the engine.