init script timeout is too short when resizing logs

Bug #1328262 reported by Tom Manville on 2014-06-09
30
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Fix Released
High
Tomislav Plavcic
5.1
Fix Released
High
Tomislav Plavcic
5.5
Fix Released
High
Tomislav Plavcic
5.6
Fix Released
High
Tomislav Plavcic

Bug Description

Currently the init script debian/percona-server-server-5.6.mysql.init starts mysqld, then pings it it until it starts. If mysql is not accessable after 14 seconds, then the init script will fail.

In MySQL 5.6 the redo logs may be resized during starup, which may take longer than 14 seconds.

In my testing (on SSDs), 30 seconds was a sufficient timeout to wait for 2GB logs to be generated.

Tags: pkg Edit Tag help

Related branches

Related bug 1072538.

tags: added: pkg

Hi,

Can you provide the exact Debian and OS versions so we can try to reproduce the same? Thanks.

Sorry, Exact Debian and PS (Percona Server) versions. i.e 5.6.*

Tom Manville (tdmanville) wrote :

We've reproed this on Ubuntu 13.10, PS 5.6.17 and Ubuntu 12.04, PS 5.6.16

Attached is our fix.

Confirmed with 5.1, 5.5 and 5.6 versions. Everywhere, 14 seconds hardcoded timeout.

/usr/bin/mysqld_safe > /dev/null 2>&1 &
     # 6s was reported in #352070 to be too few when using ndbcluster
     for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do
                sleep 1
         if mysqld_status check_alive nowarn ; then break; fi
  log_progress_msg "."
     done

Tom Manville (tdmanville) wrote :

Even 30 seconds timed out occasionally when resizing logs (crash recovery may be worse, but we haven't tested it).

We've bumped it up to 300 seconds without incident.

Vladimir Lazarenko (favoretti) wrote :

In the past we've asked Percona PS to rebuild our packages in such a way that we could override it from /etc/default/mysql, like:

xxx@db003:/var/lib/mysql/journal# cat /etc/default/mysql
# Default timeout for the server to start
MYSQL_START_TIMEOUT=180

# Default timeout for the server to stop
MYSQL_STOP_TIMEOUT=180

If this could be ported upstream - this would be awesome. We have MySQL servers with 300GB RAM, just a startup (or a shutdown for that matter) takes longer than 5 minutes. Hence, whatever value you hardcode won't cut it for edge cases - so why not make it configurable?

Thanks for the email. Making it configurable does seem like a good way to
go. If you propose this in the bug, I'll add support for the idea.

On Wed, Sep 17, 2014 at 5:21 AM, Favoretti <email address hidden> wrote:

> In the past we've asked Percona PS to rebuild our packages in such a way
> that we could override it from /etc/default/mysql, like:
>
> xxx@db003:/var/lib/mysql/journal# cat /etc/default/mysql
> # Default timeout for the server to start
> MYSQL_START_TIMEOUT=180
>
> # Default timeout for the server to stop
> MYSQL_STOP_TIMEOUT=180
>
> If this could be ported upstream - this would be awesome. We have MySQL
> servers with 300GB RAM, just a startup (or a shutdown for that matter)
> takes longer than 5 minutes. Hence, whatever value you hardcode won't
> cut it for edge cases - so why not make it configurable?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1328262
>
> Title:
> init script timeout is too short when resizing logs
>
> Status in Percona Server with XtraDB:
> Triaged
> Status in Percona Server 5.1 series:
> Triaged
> Status in Percona Server 5.5 series:
> Triaged
> Status in Percona Server 5.6 series:
> Triaged
>
> Bug description:
> Currently the init script debian/percona-server-server-5.6.mysql.init
> starts mysqld, then pings it it until it starts. If mysql is not
> accessable after 14 seconds, then the init script will fail.
>
> In MySQL 5.6 the redo logs may be resized during starup, which may
> take longer than 14 seconds.
>
> In my testing (on SSDs), 30 seconds was a sufficient timeout to wait
> for 2GB logs to be generated.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/percona-server/+bug/1328262/+subscriptions
>

Vladimir Lazarenko (favoretti) wrote :

Hey.

I did? :) or you want it as an explicit proposal?

Cheers,
Vlad

Text by me, typos by iPhone

> On 17 sep. 2014, at 19:35, Tom Manville <email address hidden> wrote:
>
> Thanks for the email. Making it configurable does seem like a good way to
> go. If you propose this in the bug, I'll add support for the idea.
>
>> On Wed, Sep 17, 2014 at 5:21 AM, Favoretti <email address hidden> wrote:
>>
>> In the past we've asked Percona PS to rebuild our packages in such a way
>> that we could override it from /etc/default/mysql, like:
>>
>> xxx@db003:/var/lib/mysql/journal# cat /etc/default/mysql
>> # Default timeout for the server to start
>> MYSQL_START_TIMEOUT=180
>>
>> # Default timeout for the server to stop
>> MYSQL_STOP_TIMEOUT=180
>>
>> If this could be ported upstream - this would be awesome. We have MySQL
>> servers with 300GB RAM, just a startup (or a shutdown for that matter)
>> takes longer than 5 minutes. Hence, whatever value you hardcode won't
>> cut it for edge cases - so why not make it configurable?
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1328262
>>
>> Title:
>> init script timeout is too short when resizing logs
>>
>> Status in Percona Server with XtraDB:
>> Triaged
>> Status in Percona Server 5.1 series:
>> Triaged
>> Status in Percona Server 5.5 series:
>> Triaged
>> Status in Percona Server 5.6 series:
>> Triaged
>>
>> Bug description:
>> Currently the init script debian/percona-server-server-5.6.mysql.init
>> starts mysqld, then pings it it until it starts. If mysql is not
>> accessable after 14 seconds, then the init script will fail.
>>
>> In MySQL 5.6 the redo logs may be resized during starup, which may
>> take longer than 14 seconds.
>>
>> In my testing (on SSDs), 30 seconds was a sufficient timeout to wait
>> for 2GB logs to be generated.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/percona-server/+bug/1328262/+subscriptions
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1328262
>
> Title:
> init script timeout is too short when resizing logs
>
> Status in Percona Server with XtraDB:
> Triaged
> Status in Percona Server 5.1 series:
> Triaged
> Status in Percona Server 5.5 series:
> Triaged
> Status in Percona Server 5.6 series:
> Triaged
>
> Bug description:
> Currently the init script debian/percona-server-server-5.6.mysql.init
> starts mysqld, then pings it it until it starts. If mysql is not
> accessable after 14 seconds, then the init script will fail.
>
> In MySQL 5.6 the redo logs may be resized during starup, which may
> take longer than 14 seconds.
>
> In my testing (on SSDs), 30 seconds was a sufficient timeout to wait
> for 2GB logs to be generated.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/percona-server/+bug/1328262/+subscriptions

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-792

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers