init script timeout is too short when resizing logs

[Tom Manville]

Currently the init script debian/percona-server-server-5.6.mysql.init starts mysqld, then pings it it until it starts. If mysql is not accessable after 14 seconds, then the init script will fail.

In MySQL 5.6 the redo logs may be resized during starup, which may take longer than 14 seconds.

In my testing (on SSDs), 30 seconds was a sufficient timeout to wait for 2GB logs to be generated.

[Laurynas Biveinis (laurynas-biveinis)]

Related bug 1072538.

[Nilnandan Joshi (nilnandan-joshi)]

Hi,

Can you provide the exact Debian and OS versions so we can try to reproduce the same? Thanks.

[Nilnandan Joshi (nilnandan-joshi)]

Sorry, Exact Debian and PS (Percona Server) versions. i.e 5.6.*

[Tom Manville (tdmanville)]

We've reproed this on Ubuntu 13.10, PS 5.6.17 and Ubuntu 12.04, PS 5.6.16

Attached is our fix.

[Nilnandan Joshi (nilnandan-joshi)]

Confirmed with 5.1, 5.5 and 5.6 versions. Everywhere, 14 seconds hardcoded timeout.

/usr/bin/mysqld_safe > /dev/null 2>&1 &
     # 6s was reported in #352070 to be too few when using ndbcluster
     for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14; do
                sleep 1
         if mysqld_status check_alive nowarn ; then break; fi
  log_progress_msg "."
     done

[Tom Manville (tdmanville)]

Even 30 seconds timed out occasionally when resizing logs (crash recovery may be worse, but we haven't tested it).

We've bumped it up to 300 seconds without incident.

[Vladimir Lazarenko (favoretti)]

In the past we've asked Percona PS to rebuild our packages in such a way that we could override it from /etc/default/mysql, like:

xxx@db003:/var/lib/mysql/journal# cat /etc/default/mysql
# Default timeout for the server to start
MYSQL_START_TIMEOUT=180

# Default timeout for the server to stop
MYSQL_STOP_TIMEOUT=180

If this could be ported upstream - this would be awesome. We have MySQL servers with 300GB RAM, just a startup (or a shutdown for that matter) takes longer than 5 minutes. Hence, whatever value you hardcode won't cut it for edge cases - so why not make it configurable?

[Tom Manville (tdmanville)]

Thanks for the email. Making it configurable does seem like a good way to
go. If you propose this in the bug, I'll add support for the idea.

On Wed, Sep 17, 2014 at 5:21 AM, Favoretti <email address hidden> wrote:

> In the past we've asked Percona PS to rebuild our packages in such a way
> that we could override it from /etc/default/mysql, like:
>
> xxx@db003:/var/lib/mysql/journal# cat /etc/default/mysql
> # Default timeout for the server to start
> MYSQL_START_TIMEOUT=180
>
> # Default timeout for the server to stop
> MYSQL_STOP_TIMEOUT=180
>
> If this could be ported upstream - this would be awesome. We have MySQL
> servers with 300GB RAM, just a startup (or a shutdown for that matter)
> takes longer than 5 minutes. Hence, whatever value you hardcode won't
> cut it for edge cases - so why not make it configurable?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1328262
>
> Title:
> init script timeout is too short when resizing logs
>
> Status in Percona Server with XtraDB:
> Triaged
> Status in Percona Server 5.1 series:
> Triaged
> Status in Percona Server 5.5 series:
> Triaged
> Status in Percona Server 5.6 series:
> Triaged
>
> Bug description:
> Currently the init script debian/percona-server-server-5.6.mysql.init
> starts mysqld, then pings it it until it starts. If mysql is not
> accessable after 14 seconds, then the init script will fail.
>
> In MySQL 5.6 the redo logs may be resized during starup, which may
> take longer than 14 seconds.
>
> In my testing (on SSDs), 30 seconds was a sufficient timeout to wait
> for 2GB logs to be generated.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/percona-server/+bug/1328262/+subscriptions
>

[Vladimir Lazarenko (favoretti)]

Hey.

I did? :) or you want it as an explicit proposal?

Cheers,
Vlad

Text by me, typos by iPhone

> On 17 sep. 2014, at 19:35, Tom Manville <email address hidden> wrote:
>
> Thanks for the email. Making it configurable does seem like a good way to
> go. If you propose this in the bug, I'll add support for the idea.
>
>> On Wed, Sep 17, 2014 at 5:21 AM, Favoretti <email address hidden> wrote:
>>
>> In the past we've asked Percona PS to rebuild our packages in such a way
>> that we could override it from /etc/default/mysql, like:
>>
>> xxx@db003:/var/lib/mysql/journal# cat /etc/default/mysql
>> # Default timeout for the server to start
>> MYSQL_START_TIMEOUT=180
>>
>> # Default timeout for the server to stop
>> MYSQL_STOP_TIMEOUT=180
>>
>> If this could be ported upstream - this would be awesome. We have MySQL
>> servers with 300GB RAM, just a startup (or a shutdown for that matter)
>> takes longer than 5 minutes. Hence, whatever value you hardcode won't
>> cut it for edge cases - so why not make it configurable?
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1328262
>>
>> Title:
>> init script timeout is too short when resizing logs
>>
>> Status in Percona Server with XtraDB:
>> Triaged
>> Status in Percona Server 5.1 series:
>> Triaged
>> Status in Percona Server 5.5 series:
>> Triaged
>> Status in Percona Server 5.6 series:
>> Triaged
>>
>> Bug description:
>> Currently the init script debian/percona-server-server-5.6.mysql.init
>> starts mysqld, then pings it it until it starts. If mysql is not
>> accessable after 14 seconds, then the init script will fail.
>>
>> In MySQL 5.6 the redo logs may be resized during starup, which may
>> take longer than 14 seconds.
>>
>> In my testing (on SSDs), 30 seconds was a sufficient timeout to wait
>> for 2GB logs to be generated.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/percona-server/+bug/1328262/+subscriptions
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1328262
>
> Title:
> init script timeout is too short when resizing logs
>
> Status in Percona Server with XtraDB:
> Triaged
> Status in Percona Server 5.1 series:
> Triaged
> Status in Percona Server 5.5 series:
> Triaged
> Status in Percona Server 5.6 series:
> Triaged
>
> Bug description:
> Currently the init script debian/percona-server-server-5.6.mysql.init
> starts mysqld, then pings it it until it starts. If mysql is not
> accessable after 14 seconds, then the init script will fail.
>
> In MySQL 5.6 the redo logs may be resized during starup, which may
> take longer than 14 seconds.
>
> In my testing (on SSDs), 30 seconds was a sufficient timeout to wait
> for 2GB logs to be generated.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/percona-server/+bug/1328262/+subscriptions

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-792