[Hardy SRU] dash bug causes mysqld_safe to spin at 100% CPU

[Jakob Unterwurzacher]

Symptoms are: After mysql server upgrades + restarts the old mysqld_safe keeps running and consumes 100% cpu, strace shows:
wait4(4294967295, 0x7fffb3b3d59c, 0, NULL) = -1 ECHILD (No child ...
wait4(4294967295, 0x7fffb3b3d59c, 0, NULL) = -1 ECHILD (No child ...
wait4(4294967295, 0x7fffb3b3d59c, 0, NULL) = -1 ECHILD (No child ...
For more info see the dupes, particularily bug 105457. Does not happen always, for reliable test case see below.

This is a bug (race condition) in dash, partly fixed by http://git.kernel.org/?p=utils/dash/dash.git;a=commit;h=3800d4934391b144fd261a7957aea72ced7d47ea . This fixes the test case and mysqld_safe, but introduces another regression: http://eeek.borgchat.net/lists/dash/msg00268.html (patch for that in the thread)

TEST CASE: Script in the commit message (link above). Allows to reproduce this always, even on Karmic still.

(Proposed fix of using #!/bin/bash instead of #!/bin/sh is not as straigtforward as expected. bash called as sh goes into sh compatibility mode, called as bash it may behave subtle differently)

NOTES: mysqld_safe was replaced by an upstart job in Lucid. But Hardy has support for 3 more years, justifiying a fix.

[Chuck Short (zulcss)]

Thanks for the patch, can you follow the process outlined by:

https://wiki.ubuntu.com/StableReleaseUpdates

thanks
chuck

[Jilles Tjoelker (jilles)]

Be sure to verify that the script works with bash's non-POSIX mode as well. A not so well known feature of bash is that it enters POSIX mode when called as "sh". This causes various subtle differences. Therefore, a #!/bin/sh script with /bin/sh -> bash might behave differently from a #!/bin/bash script.

[Martin Pitt (pitti)]

Please get this fixed in natty first (or at least alongside). The patch looks okay as a minimal workaround, so please get it uploaded.

[Jakob Unterwurzacher (jakobunt)]

Thing is, mysqld_safe isn't used anymore since Lucid as upstart has taken over (the file is still there though).

[Kristian Nielsen (knielsen)]

I fixed this for mariadb by adding a patch to the .deb packaging that puts #!/bin/bash in the header of mysqld_safe.

Fix will be available from 5.1.52 and 5.2.4.

[Kristian Nielsen (knielsen)]

Fixed in MariaDB 5.2.4 and 5.1.53

[Raghavendra D Prabhu (raghavendra-prabhu)]

I tested the testcase in the commit on Precise and there seem to no issue there
(probably fixed there).

Also, it is not the right approach to change #!/bin/sh to #!/bin/bash since
former invokes POSIX compatibility mode and if dash has problems on any
platform, then that should be fixed rather than changing it in mysqld_safe or
#!/bin/sh needs to pointed elsewhere (or with update-alternatives).

Regarding the init scripts, I see that Percona Server is not packaged with
upstart script (hence vulnerable to this if on a platform with a buggy dash) as
mysql-server-55 does. I will open a separate issue to add that.

[Raghavendra D Prabhu (raghavendra-prabhu)]

lp:1053183 is the issue for providing upstart scripts with percona-server.

[Gabriel Ramirez (gabriel1109)]

Hello, we apologize for the delayed response, but as Ubuntu has moved on from upstart to systemd, it will not affect newer releases in the same way so we will close this one out

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-1164