race condition on shutdown (leads to corrupted fs)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
mysql-5.1 (Ubuntu) |
Invalid
|
High
|
Clint Byrum | ||
Oneiric |
Invalid
|
Undecided
|
Unassigned | ||
Precise |
Invalid
|
High
|
Clint Byrum | ||
mysql-5.5 (Ubuntu) |
Fix Released
|
High
|
Clint Byrum | ||
Oneiric |
Invalid
|
Undecided
|
Unassigned | ||
Precise |
Fix Released
|
High
|
Clint Byrum | ||
sysvinit (Ubuntu) |
Fix Released
|
High
|
Clint Byrum | ||
Oneiric |
Fix Released
|
Undecided
|
Clint Byrum | ||
Precise |
Fix Released
|
High
|
Clint Byrum |
Bug Description
== SRU JUSTIFICATION ==
IMPACT: potential data loss or extension of downtime. MySQL, for example, if sent a SIGKILL before it is done flushing its buffers into MyISAM tables, will lose that data. If using InnoDB, the transactions will have to be replayed from the transaction log at startup, which can take far longer than completing the flush procedure which the 300 second kill timeout in its job file allows for.
TEST CASE:
1. create a script, /usr/local/
##### BEGIN COPY/PASTE #####
#!/usr/bin/python
import time
import signal
import logging
import sys
logging.
def shutdown_
logging.
time.sleep(15)
logging.info("now exitting...")
sys.exit(0)
signal.
logging.
while True:
time.sleep(1)
##### END COPY/PASTE #####
chmod +x /usr/local/
2. Create an upstart job file, /etc/init/
##### BEGIN COPY/PASTE #####
start on runlevel [2345]
stop on runlevel [016]
respawn
kill timeout 17
console output
exec /usr/local/
##### END COPY/PASTE #####
3. sudo initctl start 15sec
4. sudo shutdown -h now
On an affected system, the job will be sent SIGKILL before the 15 second kill timeout, so your shutdown log will look something like this:
Checking for running unattended-
TEST: 2011-12-13 01:02:54,638: sleeping 15 seconds...
* Asking all remaining processes to terminate... TEST: 2011-12-13 01:02:54,818: now exitting...
TEST: 2011-12-13 01:02:54,819: sleeping 15 seconds...
* Killing all remaining processes... [fail]
* Deconfiguring network interfaces... [ OK ]
* Deactivating swap... [ OK ]
* Will now halt
[ 68.020383] System halted.
An unaffected system will look like this:
Checking for running unattended-
TEST: 2011-12-13 00:52:30,476: sleeping 15 seconds...
* Asking all remaining processes to terminate... [ OK ]
TEST: 2011-12-13 00:52:45,497: now exitting...
* All processes ended within 16 seconds.... [ OK ]
* Deconfiguring network interfaces... [ OK ]
* Deactivating swap... [ OK ]
* Will now halt
[ 356.481556] System halted.
Note that the 15sec job is waited on once the bug is fixed, where in the unpatched version it is killed immediately.
DEV FIX: The sendsigs script has not been changed in precise other than for this patch.
REGRESSION POTENTIAL: There may be scenarios and jobs that have very high kill timeouts which will cause system shutdowns to wait for up to 300 seconds instead of the pervious 10. This is considered a good balance between waiting long enough for any reasonable application to flush its buffers and short enough that we won't run up against any battery backup systems running out of battery power.
======
I'm using mysql-server-5.1 on a 10.04 LTS installation.
The mysql db is around 27GB and on a separate partition mounted as /var/lib/mysql.
On shutdown I get the following error message:
Checking for running unattended-
[80G
[74G[ OK ]
* All processes ended within 1 seconds....
[80G
[74G[ OK ]
* Deconfiguring network interfaces...
[80G
[74G[ OK ]
* Deactivating swap...
[80G
[74G[ OK ]
* Unmounting local filesystems...
[80G umount2: Device or resource busy
umount: /var/lib/mysql: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount2: Device or resource busy
umount2: Device or resource busy
umount: /tmp: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
umount2: Device or resource busy
[74G[
[31mfail
[39;49m]
mount: / is busy
* Will now restart
[ 3369.429751] Restarting system.
On the next reboot the file system is corrupt and need to be fsck-ed.
I think the problem is, that mysql uses an upstart job (/etc/init/
stop on runlevel [016]
The rc.conf job is also triggered on runlevel 0 and 6, so they basically run at the same time.As
When /etc/rc0.
As my mysqld process takes some time to shutdown, S40umountfs and S60umountroot are run before the mysqld has quit.
Leading to the fs not being properly unmounted. It is event possible that mysqld is forcefully killed by halt in S90halt if it hasn't stopped by then.
This is a serious issue, as it can (and will) lead to data loss.
Other upstart jobs, like rsyslog.conf, use the same "stop on runlevel [016]" stanza, so they are probably affected too.
ProblemType: Bug
DistroRelease: Ubuntu 10.10
Package: mysql-server-5.1 5.1.49-1ubuntu8.1
Uname: Linux 2.6.32-5-686 i686
NonfreeKernelMo
Architecture: i386
Date: Fri Dec 10 13:41:52 2010
ProcEnviron:
PATH=(custom, no user)
LANG=de_DE.utf8
SHELL=/bin/bash
SourcePackage: mysql-5.1
Related branches
- Steve Langasek: Pending requested
-
Diff: 138 lines (+65/-29)3 files modifieddebian/changelog (+14/-0)
debian/src/initscripts/etc/init.d/sendsigs (+23/-1)
debian/src/initscripts/etc/init.d/umountroot (+28/-28)
tags: | added: patch |
Changed in sysvinit (Ubuntu): | |
importance: | Undecided → High |
assignee: | nobody → Canonical Foundations Team (canonical-foundations) |
Changed in mysql-5.5 (Ubuntu): | |
status: | New → In Progress |
importance: | Undecided → High |
assignee: | nobody → Clint Byrum (clint-fewbar) |
Changed in mysql-5.1 (Ubuntu): | |
status: | In Progress → Triaged |
Changed in mysql-5.5 (Ubuntu Precise): | |
status: | In Progress → Invalid |
Changed in mysql-5.1 (Ubuntu Precise): | |
status: | Triaged → Invalid |
Changed in sysvinit (Ubuntu Precise): | |
status: | Triaged → In Progress |
assignee: | Canonical Foundations Team (canonical-foundations) → Clint Byrum (clint-fewbar) |
Changed in mysql-5.1 (Ubuntu Oneiric): | |
status: | New → Invalid |
Changed in mysql-5.5 (Ubuntu Oneiric): | |
status: | New → Invalid |
Changed in sysvinit (Ubuntu Oneiric): | |
status: | New → In Progress |
assignee: | nobody → Clint Byrum (clint-fewbar) |
What would be the general approach to express "shut down on runlevel 0/1/6 before the disks go away" in terms of upstart triggers? Once there's an approach, pleaes hand over to canonical-server. Thanks!