MySQL launch fails silently if < 4MB of disk space is available

[Kurt Huwig]

[Impact]
MySQL will not launch if less than 4MB of disk space is available:

/etc/init/mysql.conf:
LC_ALL=C BLOCKSIZE= df --portability /var/lib/mysql/. | tail -n 1 | awk '{ exit ($4<4096) }'

This behaviour is OK, but it should not stop silently but report an error message so that the admin knows why it did not start.

[Test Case]
1) Prep system to have 4MB disk space
2) Attempt to start mysql server
3) Silent fail with upstart

[Regression Potentional]
Possible regressions may include mysql failing to start. As the upstart job has been modified.

[Adam Stokes (adam-stokes)]

This looks to be addressed already in mysql-5.5_5.5.29-0ubuntu0.12.04.2, could the OP please test with latest mysql-5.5 on Precise?

Thanks
Adam

[Kurt Huwig (k-huwig-f)]

I cannot see a difference with

ii mysql-server-5.5 5.5.29-0ubuntu0.12.04.2 MySQL database server binaries and system database setup

As I have too much disk space on my test machine, I've changed the code to check for much more space but left the rest untouched:

    LC_ALL=C BLOCKSIZE= df --portability /var/lib/mysql/. | tail -n 1 | awk '{ exit ($4<409600000) }'

The result is that the start of mysql fails, but there are no new logentries (I truncated all files before starting mysql):

root@mysql:~# service mysql start
start: Job failed to start
root@mysql:~# ls -l /var/log/mysql/error.log /var/log/mysql* /var/log/syslog
-rw-r----- 1 mysql adm 0 Apr 18 08:42 /var/log/mysql.err
-rw-r----- 1 mysql adm 0 Apr 18 08:42 /var/log/mysql.log
-rw-rw---- 1 mysql adm 0 Apr 18 08:45 /var/log/mysql/error.log
-rw-r----- 1 syslog adm 0 Apr 18 08:44 /var/log/syslog

/var/log/mysql:
total 0
-rw-rw---- 1 mysql adm 0 Apr 18 08:45 error.log

[Adam Stokes (adam-stokes)]

re #4,

Yea I realized this was in sysv script but not upstart, attached debdiffs to fix that.

Thanks for your help
Adam

[James Hunt (jamesodhunt)]

Thanks Adam, however these diffs are making use of LSB functions (from /lib/lsb/init-functions), but that script is not being sourced.

Using echo directly to stderr should be sufficient in this context (or maybe defining a log_failure_msg() function locally to do that).

[Adam Stokes (adam-stokes)]

Ok that sounds good, I'll re-work the patch this week and re-post when I have something for you.

Thanks
Adam

[James Hunt (jamesodhunt)]

Untested, but looks reasonable. However, note that you can now use the Upstart apparmor profile rather than using the pre-start for that:

    apparmor load /etc/apparmor.d/usr.sbin.mysqld

   pre-start script

       # ...

   end script

[Brian Murray (brian-murray)]

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.32-0ubuntu0.13.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.32-0ubuntu0.12.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.32-0ubuntu0.12.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Whoopie (whoopie79)]

Startup of mysqld fails here with this patch applied, I had to revert it.

[Whoopie (whoopie79)]

Where is mysqld_get_param defined or sourced from?

[Michael Bienia (geser)]

There seem to be some problems with this update:
see bug 1210380 for the precise update
see bug 1210383 for the raring update

[Michael Bienia (geser)]

and see bug 1210427 for the quantal update

[Adam Stokes (adam-stokes)]

Updated with new diffs to include the mysqld_get_param routine

[Robie Basak (racb)]

12:55 #ubuntu-devel: <stokachu> rbasak: building and testing the mysql upgrade
                                now should hopefully be ready in the next hour
16:41 #ubuntu-devel: <stokachu> rbasak: i attempted to build mysql but it
                                continually fails on a few unittests

@Adam,

What's the current status of these patches, please? Are these ready now, or are there known problems? If ready, please could you detail what testing you've performed so that we can try and avoid a second -proposed regression? Thanks!

[Robie Basak (racb)]

I've just uploaded 5.5.32-0ubuntu3, which reverts 5.5.32-0ubuntu2. In my testing in my PPA, the test suite failed once, and on build retry succeeded. So this upload may FTBFS, but even then I think it'd be better to have something that at least used to work in -proposed for us to work from.

Since we never released the SRUs, they are presumably no longer Fix Committed.

Hopefully we should be back to square one now.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.5 - 5.5.32-0ubuntu3

---------------
mysql-5.5 (5.5.32-0ubuntu3) saucy; urgency=low

  * Revert 5.5.32-0ubuntu2, which caused a severe regression making the
    package uninstallable. See LP: #1121874 for details.
 -- Robie Basak <email address hidden> Tue, 27 Aug 2013 11:55:13 +0000

[Iain Lane (laney)]

Robie, are you looking at these patches? The bug is still open & on the sponsor queue.

</patch-pilot>

[Robie Basak (racb)]

The last time I spoke to Adam on IRC, he had been unable to test his newest patches. We concluded that we should revert the old patch for now, without replacing them with a new one, since we weren't confident about applying the new set.

I assume this is still the current status?

I think it's appropriate to remove this bug from the sponsor queue for now. Adam: please resubscribe ubuntu-sponsors when you think the patch is tested and ready.

[Adam Stokes (adam-stokes)]

Thanks and will do. I haven't been able to build a successful mysql package via sbuild with or without the patches so Ill go through the process of getting it uploaded to a PPA and built that way.

Ill re-subscribe sponsors once I can get a reliable package built.

[Dave Chiluk (chiluk)]

I also have not been able to successfully build using sbuild or pbuilder. I started from a clean 12.04.3 install, and followed the sbuild guide available here with no success https://wiki.ubuntu.com/SecurityTeam/BuildEnvironment

+mysqlcheck: Got error: 2003: Can't connect to MySQL server on 'not_existing_host' (e
rrno) when trying to connect
But after looking at the testcase, it looks like it's doing this on purpose. Either way, I give up.

So I built these packages in a ppa available here.
https://launchpad.net/~chiluk/+archive/1121874

Anyone wishing to test this new install is welcome to do so

[Dave Chiluk (chiluk)]

I'm taking this over from Adam.

I found the bug in the debdiffs from above. I pushed the necessary changes to my ppa https://launchpad.net/~chiluk/+archive/1121874

I will be testing out of that ppa tomorrow, and will post debdiffs once I'm comfortable with their quality.

[Dave Chiluk (chiluk)]

I'm attaching updated fixed debdiffs. I changed it from original for readability and functionality.

[Dave Chiluk (chiluk)]

Subscribing Ubuntu-sru for p,q,r, and ubuntu-sponsors for saucy.

[Dave Chiluk (chiluk)]

trusty debdiff

[Dave Chiluk (chiluk)]

I removed the bzr branches related to the previous change that caused the earlier regression.

I also have included updated debdiffs with a more readable solution.

[James Page (james-page)]

@Dave

Thanks for the updated debdiffs; I did a quick test and the new proposed upstart configuration looks good and tests OK.

I've rebased the trusty debdiff for the 5.5.34 release that we now have in distro and will upload shortly; Please could you rebase the others appropriately - I appreciate its like trying to hit a moving target - sorry!

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.5 - 5.5.34-0ubuntu2

---------------
mysql-5.5 (5.5.34-0ubuntu2) trusty; urgency=low

  [ Eduardo Damato ]
  * Fix upstart script to account for datadir disk shortage (LP: #1121874).
 -- Dave Chiluk <email address hidden> Mon, 18 Nov 2013 09:38:46 +0000

[Marc Deslauriers (mdeslaur)]

I've unsubscribed ubuntu-sponsors for now, since there is no actionable item.

Please re-subscribe ubuntu-sponsors once updated debdiffs have been attached. Thanks!

[Dave Chiluk (chiluk)]

I finally got some time to update these debdiffs.

[Dave Chiluk (chiluk)]

@mdeslaur
@james-page

Rebased the debdiffs against the latest mysql-5.5 in each series. I also deleted the earlier debdiffs against the earlier versions.

[Chris J Arges (arges)]

uploaded for p/q/r/s

[Brian Murray (brian-murray)]

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.34-0ubuntu0.13.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.34-0ubuntu0.12.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

I'm going to not approve the Raring upload as Raring becomes End of Life this month.

[Brian Murray (brian-murray)]

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.34-0ubuntu0.12.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ubuntu Foundations Team Bug Bot (crichton)]

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of mysql-5.5 from precise-proposed was performed and bug 1270553 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and tag 1270553 "bot-stop-nagging". Thanks!

[Robie Basak (racb)]

I don't consider bug 1270553 to be a regression in this update. There are a number of users who have broken MySQL installations due to a previous local problem that has gone unrepaired, and an updated package shows them up because an attempt by the postinst to restart the daemon consequently fails. I think bug 1270553 is such an example. I have subscribed to it in case this changes.

[Ubuntu Foundations Team Bug Bot (crichton)]

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of mysql-5.5 from precise-proposed was performed and bug 1270553 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and tag 1270553 "bot-stop-nagging". Thanks!

[Marc Deslauriers (mdeslaur)]

Unfortunately, the security updates published today superseded the packages in -proposed.

[Marc Deslauriers (mdeslaur)]

I have uploaded updated packages for -proposed for processing by the SRU team.

[Brian Murray (brian-murray)]

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.35-0ubuntu0.13.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.35-0ubuntu0.12.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.35-0ubuntu0.12.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dave Chiluk (chiluk)]

I went through and verified these tonight.

I never want to see this bug ever again.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.5 - 5.5.35-0ubuntu0.12.04.2

---------------
mysql-5.5 (5.5.35-0ubuntu0.12.04.2) precise; urgency=medium

  [ Eduardo Damato ]
  * Fix upstart script to account for datadir disk shortage (LP: #1121874)
 -- Dave Chiluk <email address hidden> Tue, 21 Jan 2014 13:11:07 -0500

[Colin Watson (cjwatson)]

The verification of the Stable Release Update for mysql-5.5 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.5 - 5.5.35-0ubuntu0.12.10.2

---------------
mysql-5.5 (5.5.35-0ubuntu0.12.10.2) quantal; urgency=medium

  [ Eduardo Damato ]
  * Fix upstart script to account for datadir disk shortage (LP: #1121874)
 -- Dave Chiluk <email address hidden> Tue, 21 Jan 2014 13:09:30 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-5.5 - 5.5.35-0ubuntu0.13.10.2

---------------
mysql-5.5 (5.5.35-0ubuntu0.13.10.2) saucy; urgency=medium

  [ Eduardo Damato ]
  * Fix upstart script to account for datadir disk shortage (LP: #1121874)
 -- Dave Chiluk <email address hidden> Tue, 21 Jan 2014 13:06:45 -0500