MySQL launch fails silently if < 4MB of disk space is available

Bug #1121874 reported by Kurt Huwig on 2013-02-11
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
mysql-5.5 (Ubuntu)
Medium
Dave Chiluk
Precise
Medium
Dave Chiluk
Quantal
Medium
Dave Chiluk
Raring
Medium
Dave Chiluk
Saucy
Medium
Dave Chiluk

Bug Description

[Impact]
MySQL will not launch if less than 4MB of disk space is available:

/etc/init/mysql.conf:
LC_ALL=C BLOCKSIZE= df --portability /var/lib/mysql/. | tail -n 1 | awk '{ exit ($4<4096) }'

This behaviour is OK, but it should not stop silently but report an error message so that the admin knows why it did not start.

[Test Case]
1) Prep system to have 4MB disk space
2) Attempt to start mysql server
3) Silent fail with upstart

[Regression Potentional]
Possible regressions may include mysql failing to start. As the upstart job has been modified.

Robie Basak (racb) on 2013-02-11
Changed in mysql-5.5 (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Robie Basak (racb) on 2013-04-02
Changed in mysql-5.5 (Ubuntu Raring):
assignee: nobody → Robie Basak (racb)
Adam Stokes (adam-stokes) wrote :

This looks to be addressed already in mysql-5.5_5.5.29-0ubuntu0.12.04.2, could the OP please test with latest mysql-5.5 on Precise?

Thanks
Adam

Kurt Huwig (k-huwig-f) wrote :

I cannot see a difference with

ii mysql-server-5.5 5.5.29-0ubuntu0.12.04.2 MySQL database server binaries and system database setup

As I have too much disk space on my test machine, I've changed the code to check for much more space but left the rest untouched:

    LC_ALL=C BLOCKSIZE= df --portability /var/lib/mysql/. | tail -n 1 | awk '{ exit ($4<409600000) }'

The result is that the start of mysql fails, but there are no new logentries (I truncated all files before starting mysql):

root@mysql:~# service mysql start
start: Job failed to start
root@mysql:~# ls -l /var/log/mysql/error.log /var/log/mysql* /var/log/syslog
-rw-r----- 1 mysql adm 0 Apr 18 08:42 /var/log/mysql.err
-rw-r----- 1 mysql adm 0 Apr 18 08:42 /var/log/mysql.log
-rw-rw---- 1 mysql adm 0 Apr 18 08:45 /var/log/mysql/error.log
-rw-r----- 1 syslog adm 0 Apr 18 08:44 /var/log/syslog

/var/log/mysql:
total 0
-rw-rw---- 1 mysql adm 0 Apr 18 08:45 error.log

Changed in mysql-5.5 (Ubuntu Precise):
importance: Undecided → Medium
Changed in mysql-5.5 (Ubuntu Quantal):
importance: Undecided → Critical
importance: Critical → Medium
Adam Stokes (adam-stokes) wrote :

re #4,

Yea I realized this was in sysv script but not upstart, attached debdiffs to fix that.

Thanks for your help
Adam

description: updated
James Hunt (jamesodhunt) wrote :

Thanks Adam, however these diffs are making use of LSB functions (from /lib/lsb/init-functions), but that script is not being sourced.

Using echo directly to stderr should be sufficient in this context (or maybe defining a log_failure_msg() function locally to do that).

Adam Stokes (adam-stokes) wrote :

Ok that sounds good, I'll re-work the patch this week and re-post when I have something for you.

Thanks
Adam

James Hunt (jamesodhunt) wrote :

Untested, but looks reasonable. However, note that you can now use the Upstart apparmor profile rather than using the pre-start for that:

    apparmor load /etc/apparmor.d/usr.sbin.mysqld

   pre-start script

       # ...

   end script

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into raring-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.32-0ubuntu0.13.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.5 (Ubuntu Raring):
status: Triaged → Fix Committed
tags: added: verification-needed
Changed in mysql-5.5 (Ubuntu Quantal):
status: New → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.32-0ubuntu0.12.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.5 (Ubuntu Precise):
status: New → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.32-0ubuntu0.12.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Whoopie (whoopie79) wrote :

Startup of mysqld fails here with this patch applied, I had to revert it.

Whoopie (whoopie79) wrote :

Where is mysqld_get_param defined or sourced from?

Michael Bienia (geser) wrote :

There seem to be some problems with this update:
see bug 1210380 for the precise update
see bug 1210383 for the raring update

Michael Bienia (geser) wrote :

and see bug 1210427 for the quantal update

Robie Basak (racb) on 2013-08-09
tags: added: verification-failed
removed: verification-needed
Adam Stokes (adam-stokes) wrote :

Updated with new diffs to include the mysqld_get_param routine

tags: removed: verification-failed
Robie Basak (racb) wrote :

12:55 #ubuntu-devel: <stokachu> rbasak: building and testing the mysql upgrade
                                now should hopefully be ready in the next hour
16:41 #ubuntu-devel: <stokachu> rbasak: i attempted to build mysql but it
                                continually fails on a few unittests

@Adam,

What's the current status of these patches, please? Are these ready now, or are there known problems? If ready, please could you detail what testing you've performed so that we can try and avoid a second -proposed regression? Thanks!

Robie Basak (racb) wrote :

I've just uploaded 5.5.32-0ubuntu3, which reverts 5.5.32-0ubuntu2. In my testing in my PPA, the test suite failed once, and on build retry succeeded. So this upload may FTBFS, but even then I think it'd be better to have something that at least used to work in -proposed for us to work from.

Since we never released the SRUs, they are presumably no longer Fix Committed.

Hopefully we should be back to square one now.

Changed in mysql-5.5 (Ubuntu Precise):
status: Fix Committed → Triaged
Changed in mysql-5.5 (Ubuntu Quantal):
status: Fix Committed → Triaged
Changed in mysql-5.5 (Ubuntu Raring):
status: Fix Committed → Triaged
assignee: Robie Basak (racb) → nobody
Changed in mysql-5.5 (Ubuntu):
assignee: Robie Basak (racb) → nobody
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.5 - 5.5.32-0ubuntu3

---------------
mysql-5.5 (5.5.32-0ubuntu3) saucy; urgency=low

  * Revert 5.5.32-0ubuntu2, which caused a severe regression making the
    package uninstallable. See LP: #1121874 for details.
 -- Robie Basak <email address hidden> Tue, 27 Aug 2013 11:55:13 +0000

Changed in mysql-5.5 (Ubuntu):
status: Triaged → Fix Released
Robie Basak (racb) on 2013-08-29
Changed in mysql-5.5 (Ubuntu):
status: Fix Released → Triaged
Iain Lane (laney) wrote :

Robie, are you looking at these patches? The bug is still open & on the sponsor queue.

</patch-pilot>

Robie Basak (racb) wrote :

The last time I spoke to Adam on IRC, he had been unable to test his newest patches. We concluded that we should revert the old patch for now, without replacing them with a new one, since we weren't confident about applying the new set.

I assume this is still the current status?

I think it's appropriate to remove this bug from the sponsor queue for now. Adam: please resubscribe ubuntu-sponsors when you think the patch is tested and ready.

Adam Stokes (adam-stokes) wrote :

Thanks and will do. I haven't been able to build a successful mysql package via sbuild with or without the patches so Ill go through the process of getting it uploaded to a PPA and built that way.

Ill re-subscribe sponsors once I can get a reliable package built.

Dave Chiluk (chiluk) wrote :

I also have not been able to successfully build using sbuild or pbuilder. I started from a clean 12.04.3 install, and followed the sbuild guide available here with no success https://wiki.ubuntu.com/SecurityTeam/BuildEnvironment

+mysqlcheck: Got error: 2003: Can't connect to MySQL server on 'not_existing_host' (e
rrno) when trying to connect
But after looking at the testcase, it looks like it's doing this on purpose. Either way, I give up.

So I built these packages in a ppa available here.
https://launchpad.net/~chiluk/+archive/1121874

Anyone wishing to test this new install is welcome to do so

Dave Chiluk (chiluk) wrote :

I'm taking this over from Adam.

I found the bug in the debdiffs from above. I pushed the necessary changes to my ppa https://launchpad.net/~chiluk/+archive/1121874

I will be testing out of that ppa tomorrow, and will post debdiffs once I'm comfortable with their quality.

Dave Chiluk (chiluk) wrote :

I'm attaching updated fixed debdiffs. I changed it from original for readability and functionality.

Dave Chiluk (chiluk) wrote :

Subscribing Ubuntu-sru for p,q,r, and ubuntu-sponsors for saucy.

Dave Chiluk (chiluk) on 2013-10-21
Changed in mysql-5.5 (Ubuntu):
status: Triaged → In Progress
Changed in mysql-5.5 (Ubuntu Precise):
status: Triaged → In Progress
Changed in mysql-5.5 (Ubuntu Quantal):
status: Triaged → In Progress
Changed in mysql-5.5 (Ubuntu Raring):
status: Triaged → In Progress
Changed in mysql-5.5 (Ubuntu Quantal):
assignee: nobody → Dave Chiluk (chiluk)
Changed in mysql-5.5 (Ubuntu Raring):
assignee: nobody → Dave Chiluk (chiluk)
Dave Chiluk (chiluk) on 2013-10-22
Changed in mysql-5.5 (Ubuntu Precise):
assignee: nobody → Dave Chiluk (chiluk)
Changed in mysql-5.5 (Ubuntu):
assignee: nobody → Dave Chiluk (chiluk)
Chris J Arges (arges) on 2013-10-22
Changed in mysql-5.5 (Ubuntu Saucy):
assignee: nobody → Dave Chiluk (chiluk)
importance: Undecided → Medium
status: New → In Progress
Dave Chiluk (chiluk) wrote :

trusty debdiff

Dave Chiluk (chiluk) wrote :
Dave Chiluk (chiluk) on 2013-10-31
description: updated
Dave Chiluk (chiluk) wrote :

I removed the bzr branches related to the previous change that caused the earlier regression.

I also have included updated debdiffs with a more readable solution.

James Page (james-page) wrote :

@Dave

Thanks for the updated debdiffs; I did a quick test and the new proposed upstart configuration looks good and tests OK.

I've rebased the trusty debdiff for the 5.5.34 release that we now have in distro and will upload shortly; Please could you rebase the others appropriately - I appreciate its like trying to hit a moving target - sorry!

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.5 - 5.5.34-0ubuntu2

---------------
mysql-5.5 (5.5.34-0ubuntu2) trusty; urgency=low

  [ Eduardo Damato ]
  * Fix upstart script to account for datadir disk shortage (LP: #1121874).
 -- Dave Chiluk <email address hidden> Mon, 18 Nov 2013 09:38:46 +0000

Changed in mysql-5.5 (Ubuntu):
status: In Progress → Fix Released
Marc Deslauriers (mdeslaur) wrote :

I've unsubscribed ubuntu-sponsors for now, since there is no actionable item.

Please re-subscribe ubuntu-sponsors once updated debdiffs have been attached. Thanks!

Dave Chiluk (chiluk) wrote :

I finally got some time to update these debdiffs.

Dave Chiluk (chiluk) wrote :
Dave Chiluk (chiluk) wrote :
Dave Chiluk (chiluk) wrote :
Dave Chiluk (chiluk) wrote :

@mdeslaur
@james-page

Rebased the debdiffs against the latest mysql-5.5 in each series. I also deleted the earlier debdiffs against the earlier versions.

Chris J Arges (arges) wrote :

uploaded for p/q/r/s

Brian Murray (brian-murray) wrote :

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.34-0ubuntu0.13.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.5 (Ubuntu Saucy):
status: In Progress → Fix Committed
tags: added: verification-needed
Brian Murray (brian-murray) wrote :

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.34-0ubuntu0.12.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in mysql-5.5 (Ubuntu Quantal):
status: In Progress → Fix Committed
Brian Murray (brian-murray) wrote :

I'm going to not approve the Raring upload as Raring becomes End of Life this month.

Changed in mysql-5.5 (Ubuntu Precise):
status: In Progress → Fix Committed
Brian Murray (brian-murray) wrote :

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.34-0ubuntu0.12.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Dave Chiluk (chiluk) on 2014-01-17
tags: added: verification-precise-done

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of mysql-5.5 from precise-proposed was performed and bug 1270553 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and tag 1270553 "bot-stop-nagging". Thanks!

tags: added: verification-failed
Robie Basak (racb) wrote :

I don't consider bug 1270553 to be a regression in this update. There are a number of users who have broken MySQL installations due to a previous local problem that has gone unrepaired, and an updated package shows them up because an attempt by the postinst to restart the daemon consequently fails. I think bug 1270553 is such an example. I have subscribed to it in case this changes.

tags: removed: verification-failed

As a part of the Stable Release Updates quality process a search for Launchpad bug reports using the version of mysql-5.5 from precise-proposed was performed and bug 1270553 was found. Please investigate this bug report to ensure that a regression will not be created by this SRU. In the event that this is not a regression remove the "verification-failed" tag from this bug report and tag 1270553 "bot-stop-nagging". Thanks!

tags: added: verification-failed
Marc Deslauriers (mdeslaur) wrote :

Unfortunately, the security updates published today superseded the packages in -proposed.

Changed in mysql-5.5 (Ubuntu Raring):
status: In Progress → Won't Fix
Dave Chiluk (chiluk) on 2014-01-21
tags: removed: verification-failed
Dave Chiluk (chiluk) on 2014-01-21
tags: added: verification-done-precise verification-done-quantal
removed: verification-precise-done
Dave Chiluk (chiluk) on 2014-01-21
tags: removed: verification-done-precise verification-done-quantal
Marc Deslauriers (mdeslaur) wrote :

I have uploaded updated packages for -proposed for processing by the SRU team.

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.35-0ubuntu0.13.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Brian Murray (brian-murray) wrote :

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into quantal-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.35-0ubuntu0.12.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Brian Murray (brian-murray) wrote :

Hello Kurt, or anyone else affected,

Accepted mysql-5.5 into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/mysql-5.5/5.5.35-0ubuntu0.12.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Dave Chiluk (chiluk) on 2014-01-21
tags: added: verification-done-trusty
Dave Chiluk (chiluk) on 2014-01-21
tags: added: verification-done-quantal
Dave Chiluk (chiluk) wrote :

I went through and verified these tonight.

I never want to see this bug ever again.

tags: added: verification-done verification-done-precise verification-done-saucy
removed: verification-needed
Changed in mysql-5.5 (Ubuntu Precise):
status: Fix Committed → Fix Released
Changed in mysql-5.5 (Ubuntu Saucy):
status: Fix Committed → Fix Released
Changed in mysql-5.5 (Ubuntu Quantal):
status: Fix Committed → Fix Released
Chris J Arges (arges) on 2014-01-22
Changed in mysql-5.5 (Ubuntu Precise):
status: Fix Released → Fix Committed
Changed in mysql-5.5 (Ubuntu Saucy):
status: Fix Released → Fix Committed
Changed in mysql-5.5 (Ubuntu Quantal):
status: Fix Released → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.5 - 5.5.35-0ubuntu0.12.04.2

---------------
mysql-5.5 (5.5.35-0ubuntu0.12.04.2) precise; urgency=medium

  [ Eduardo Damato ]
  * Fix upstart script to account for datadir disk shortage (LP: #1121874)
 -- Dave Chiluk <email address hidden> Tue, 21 Jan 2014 13:11:07 -0500

Changed in mysql-5.5 (Ubuntu Precise):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for mysql-5.5 has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.5 - 5.5.35-0ubuntu0.12.10.2

---------------
mysql-5.5 (5.5.35-0ubuntu0.12.10.2) quantal; urgency=medium

  [ Eduardo Damato ]
  * Fix upstart script to account for datadir disk shortage (LP: #1121874)
 -- Dave Chiluk <email address hidden> Tue, 21 Jan 2014 13:09:30 -0500

Changed in mysql-5.5 (Ubuntu Quantal):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package mysql-5.5 - 5.5.35-0ubuntu0.13.10.2

---------------
mysql-5.5 (5.5.35-0ubuntu0.13.10.2) saucy; urgency=medium

  [ Eduardo Damato ]
  * Fix upstart script to account for datadir disk shortage (LP: #1121874)
 -- Dave Chiluk <email address hidden> Tue, 21 Jan 2014 13:06:45 -0500

Changed in mysql-5.5 (Ubuntu Saucy):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers