Percona Server with XtraDB

slow XtraDB shutdown due to 10 second sleep in purge thread

Reported by Kristian Nielsen on 2010-09-20
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Maria
Fix Released
Low
Kristian Nielsen
Percona Server
Low
Alexey Kopytov

Bug Description

Shutting down XtraDB can take up to 10 seconds, even when there is no actual
work to do during shutdown.

An easy way to see this is to run a quick innodb test in the mysql-test-run.pl
test suite:

    ./mtr innodb-consistent

This takes >10 seconds, even though the actual test runs instantaneous.

The problem is this code in srv/srv0srv.c:

    srv_purge_thread(...)
    {
        ulint sleep_ms= 10000; /* initial: 10 sec. */
        ...

    loop:
        if (srv_shutdown_state > 0) {
            ...
        }
        os_thread_sleep( sleep_ms * 1000 );
        ...
        goto loop;

So right after startup, the purge thread sleeps for 10 seconds, so server
cannot shut down for at least 10 seconds after startup.

Of course, in a production server, the purge thread sleep could have adjusted
itself to less than 10 seconds, and a busy server anyway will probably need
much more than 10 seconds to shut down cleanly.

But for development and testing, it is annoying to have to wait for 10 seconds
for shutdown.

Suggested fix:

Change the os_thread_sleep() into a sleep that can be interrupted by
shutdown. For example a wait with os_event_wait_time() on an event created
specifically for this. (Note that currently os_event_wait_time() waits
infinitely on non-windows, but it looks easy to fix this using
pthread_cond_timedwait()). Then signal the purge thread sleep to wakeup early
during server shutdown.

Related branches

Kristian Nielsen (knielsen) wrote :

Work-around: --innodb-fast-shutdown=2, if that is acceptable (which it probably often is during test/development).

Kristian Nielsen (knielsen) wrote :

Just tested running the full test suite on my laptop (release build):

(cd mysql-test && time ./mtr --mem --parallel=3 --force)
    -> 18:16.60elapsed

(cd mysql-test && time ./mtr --mem --parallel=3 --force --mysqld=--loose-innodb-fast-shutdown=2)
    -> 11:14.47elapsed

So that's 38% faster ... so fixing this does seem rather worth-while.

Kristian Nielsen (knielsen) wrote :

Here is a patch for this.

It greatly speeds up server shutdown for tests (though --innodb-fast-shutdown=2 is still a couple seconds faster).

Changed in maria:
status: New → Fix Committed
assignee: nobody → Kristian Nielsen (knielsen)
importance: Undecided → Low
Kristian Nielsen (knielsen) wrote :

I updated the patch, attached. I will push this patch into MariaDB, hopefully it can be included
upstream in XtraDB.

Kristian Nielsen (knielsen) wrote :

I used PMP to find some more sleeps in the XtraDB shutdown code. This additional
patch, which goes on top of the one attached before, fixes the remaining cases.

With this, server shutdown during mtr tests is very close to --innodb-fast-shutdown-2.

Changed in percona-server:
status: New → Confirmed
importance: Undecided → Low
assignee: nobody → Alexey Kopytov (akopytov)
milestone: none → 5.1-13.0
Kristian Nielsen (knielsen) wrote :

Attaching BSD copyright license for the patches submitted to this bug report.

Changed in percona-server:
status: Confirmed → Fix Committed
Sergei (sergii) on 2010-11-30
Changed in maria:
milestone: none → 5.1
Changed in percona-server:
status: Fix Committed → Fix Released
Changed in maria:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers