mysql timeoutsec results in killing mysql process
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
mysql-5.7 (Ubuntu) |
Fix Released
|
Undecided
|
Paride Legovini | ||
Bionic |
Fix Released
|
Undecided
|
Paride Legovini | ||
mysql-8.0 (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Eoan |
Won't Fix
|
Medium
|
Unassigned | ||
Focal |
Fix Released
|
Medium
|
Paride Legovini |
Bug Description
[Impact]
mysql-server-
Very large databases can exceed the 600s timeout, and a safe tradeoff between timing out at some point and waiting long enough to accommodate large/huge databases does not seem to exist.
This issue has been fixed in Debian and in Ubuntu >=Hirsute by disabling the timeout (TimeoutSec=
[Test Plan]
This is probably the most interesting bit of the SRU :-)
In order to test this for real, as opposed to testing systemd's TimeoutSec, we need to make mysql very slow when it loads its tables. One way it to actually have huge tables, but I have no idea on how big they'd need to be. The other way is to slow down access to /var/lib/mysql at the I/O level. Something on these lines:
apt install mysql-server-8.0
systemctl stop mysql
cd /var/lib
mv mysql mysql.bak
truncate -s 300M mysql.blk
losetup --show --find mysql.blk
dmsetup create slowdev --table \
"0 $(blockdev --getsz /dev/loopX) delay /dev/loopX 0 100"
# With /dev/loopX as printed by losetup, 100 = 100ms r/w delay
# See: https:/
mkfs.ext4 /dev/mapper/slowdev
mkdir mysql
mount /dev/mapper/slowdev mysql
chown mysql:mysql mysql
cp -av mysql.bak/* mysql
time systemctl start mysql # slow!
systemctl status mysql
By tuning the delay parameter it is possible to trigger the timeout with the pre-SRU package, and verify that post-SRU it can load in more than 10 minutes.
Note: this can be tested in an LXD VM but it requires booting the linux-image-generic kernel, as linux-image-kvm doesn't ship the dm delay target.
The lxd-agent won't work, just ssh-import-id and ssh in.
(I think this is overkill for this SRU, but it's a quite general way to make stuff slow. I'm sure it will come useful in other cases!)
[Where problems could occur]
The TimeoutSec=infinity syntax is supported by the systemd versions in all the supported releases of Ubuntu, so this won't be a problem.
Then only change in behavior due to this change will happen on systems where the timeout is reached, and mysql is thus killed. In these cases the database server wouldn't be running in any case, but there could be cases of bad or overgrown databases (e.g. because of a runaway script adding infinite data) where the timeout is doing the right thing, preventing mysql from consuming system resources forever. In these already broken systems TimeoutSec=infinity may increase the breakage. This won't affect working production systems.
[Development Fix]
[Stable Fix]
The same fix is already already landed in Hirsute, Impish and Debian unstable.
[Original Description]
MySQL on 20.04 has TimeoutSec set to 600 (IIRC) in the systemd script. This has the effect of killing the MySQL process if this timeout is reached.
IMHO this is a Very Bad Idea. A database server process should only be force killed by a user action.
I would prefer that the server had unlimited time to cleanly shutdown and startup (eg if recovering).
Our DB is about 500GB with some very large tables (for us at least) eg. 250GB and we've had more than a few unfortunate delays as a result of delayed startup caused by recoveries because MySQL was killed prematurely.
Because MySQL 8.0 has reduced the default logging level, it was not clear to me that the process was being force killed.
I believe the MySQL team are of the same view as me per https:/
```
[12 Jul 2019 15:57] Paul Dubois
Posted by developer:
Fixed in 8.0.18.
On Debian, long InnoDB recovery times at startup could cause systemd
service startup failure. The default systemd service timeout is now
disabled (consistent with RHEL) to prevent this from happening.
```
Related branches
- Utkarsh Gupta (community): Approve
- Canonical Server Core Reviewers: Pending requested
- Canonical Server: Pending requested
-
Diff: 30 lines (+10/-1)2 files modifieddebian/changelog (+9/-0)
debian/mysql-server-8.0.mysql.service (+1/-1)
- Utkarsh Gupta (community): Approve
- Canonical Server Core Reviewers: Pending requested
- Canonical Server: Pending requested
-
Diff: 30 lines (+10/-1)2 files modifieddebian/changelog (+9/-0)
debian/mysql-server-5.7.mysql.service (+1/-1)
affects: | xorg-server (Ubuntu) → mysql-8.0 (Ubuntu) |
tags: | added: focal |
no longer affects: | mysql-8.0 (Ubuntu Groovy) |
Changed in mysql-8.0 (Ubuntu Focal): | |
status: | New → Triaged |
Changed in mysql-8.0 (Ubuntu Eoan): | |
status: | New → Triaged |
Changed in mysql-8.0 (Ubuntu Bionic): | |
status: | New → Triaged |
importance: | Undecided → Medium |
Changed in mysql-8.0 (Ubuntu Eoan): | |
importance: | Undecided → Medium |
Changed in mysql-8.0 (Ubuntu Focal): | |
importance: | Undecided → Medium |
Changed in mysql-8.0 (Ubuntu): | |
importance: | Undecided → High |
tags: | added: server-next |
tags: | added: bitesize |
description: | updated |
Changed in mysql-8.0 (Ubuntu Bionic): | |
assignee: | nobody → Paride Legovini (paride) |
Changed in mysql-8.0 (Ubuntu Focal): | |
assignee: | nobody → Paride Legovini (paride) |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
no longer affects: | mysql-5.7 (Ubuntu Eoan) |
no longer affects: | mysql-5.7 (Ubuntu Focal) |
no longer affects: | mysql-5.7 (Ubuntu Bionic) |
Changed in mysql-5.7 (Ubuntu): | |
status: | New → Triaged |
no longer affects: | mysql-8.0 (Ubuntu Bionic) |
Changed in mysql-5.7 (Ubuntu): | |
assignee: | nobody → Paride Legovini (paride) |
description: | updated |
Changed in mysql-5.7 (Ubuntu Bionic): | |
assignee: | nobody → Paride Legovini (paride) |
I see I have created this bug report in xorg instead of MySQL. Please accept my apology and either move or delete. I am happy to recreate, just tell me which.