Mysql server take 10 minutes to stop without message after time drift
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| mysql-5.7 (Ubuntu) |
Low
|
Norvald H. Ryeng |
Bug Description
Hello,
I figure out an issue and it took time to understand what happened:
* hardware clock is 2 hours in future
* during system boot: mysql start at the same time than ntp
* ntp fix clock which does a drift of 2 hours in past
Now, stopping mysql with “service mysql stop”:
* take 10 minutes
* provides no no information of what's happening to the user
* process is killed by the way (no clean stop)
To solve this issue, mysql should be started after ntp.
Distributor ID: Ubuntu
Description: Ubuntu 16.04 LTS
Release: 16.04
Codename: xenial
Here are the logs:
root@xenial:~# LANG=C journalctl -u mysql.service
-- Logs begin at Fri 2016-07-08 14:33:07 CEST, end at Fri 2016-07-08 11:46:00 CEST. --
Jul 08 14:34:11 xenial systemd[1]: Starting MySQL Community Server...
Jul 08 14:34:12 xenial systemd[1]: Started MySQL Community Server.
Jul 08 11:36:00 xenial systemd[1]: Stopping MySQL Community Server...
Jul 08 11:46:00 xenial systemd[1]: mysql.service: State 'stop-sigterm' timed out. Killing.
Jul 08 11:46:00 xenial systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL
Jul 08 11:46:00 xenial systemd[1]: Stopped MySQL Community Server.
Jul 08 11:46:00 xenial systemd[1]: mysql.service: Unit entered failed state.
Jul 08 11:46:00 xenial systemd[1]: mysql.service: Failed with result 'signal'.
root@xenial:~# cat /var/log/
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
2016-07-
Lars Tangvald (lars-tangvald) wrote : | #1 |
Robie Basak (racb) wrote : | #2 |
I'm not sure it's reasonable to adjust service startups. Following that logic, every package would have to start after ntp.service, and there should be a more general solution to that problem.
Can MySQL be adjusted to be a little more robust to the time changing when it is idle? I understand that in the general case this isn't really solvable though, which is exactly why ntp skews time. It is essential to have the system time on a server correct, and I don't think every case of things screwing up are reasonable to fix. It is reasonable to improve robustness if a fix is trivial, though.
Changed in mysql-5.7 (Ubuntu): | |
importance: | Undecided → Low |
Launchpad Janitor (janitor) wrote : | #3 |
Status changed to 'Confirmed' because the bug affects multiple users.
Changed in mysql-5.7 (Ubuntu): | |
status: | New → Confirmed |
mateddy (mateddy) wrote : | #4 |
I go this problem, however, adding After=ntp.service wouldn't fix it.
Changed in mysql-5.7 (Ubuntu): | |
assignee: | nobody → Norvald H. Ryeng (nryeng) |
I also have this problem with Ubuntu 16.04.2 LTS:
| $ lsb_release --all
| No LSB modules are available.
| Distributor ID: Ubuntu
| Description: Ubuntu 16.04.2 LTS
| Release: 16.04
| Codename: xenial
|
| $ dmidecode -s system-product-name
| VirtualBox
After a reboot the activation time of MySQL is one hour ahead.
1. If I wait one hour after reboot, I can shutdown MySQL instantly:
| $ systemctl status mysql.service
| ...
| Active: active (running) since Fri 2017-02-10 18:20:08 CET; 59min left
| ...
|
| $ date
| Fri Feb 10 17:20:54 CET 2017
|
| $ sleep 60m
|
| $ date
| Fri Feb 10 18:21:02 CET 2017
|
| $ systemctl status mysql.service
| ...
| Active: active (running) since Fri 2017-02-10 18:20:08 CET; 57s ago
| ...
|
| $ time systemctl stop mysql.service
|
| real 0m4.928s
| user 0m0.028s
| sys 0m0.008s
2. If I try it earlier, the shutdown fails after 10 minutes and MySQL gets killed by systemd with SIGKILL.
3. If I increase the systemd timeout of MySQL to two hours, then the shutdown successfully ends after one hour. The attached error-20170210.log contains this case and also the time jump:
| 2017-02-
| ...
| 2017-02-
4. As suggested, I tried to start MySQL after the ntp.service, but it didn't helped:
| $ apt-get install ntp
|
| $ grep ^After /lib/systemd/
| After=ntp.service
|
| $ systemctl show mysql.service | grep ^After
| After=ntp.service basic.target system.slice systemd-
5. I also tried to start MySQL after the systemd-
| $ cat /lib/systemd/
| ...
| ConditionFileIs
| ...
|
| $ dpkg --purge ntp
|
| $ grep ^After /lib/systemd/
| After=systemd-
|
| $ systemctl show mysql.service | grep ^After
| After=systemd-
6. Is there anything else I can try or do?
7. Maybe the importance of this issue should be raised, because killing MySQL during its shutdown with SIGKILL feels a bit risky.
I solved my above time jump problem by installing the VirtualBox guest additions as described here: https:/
f10w (huudienkhue-le) wrote : | #8 |
Any fix to this problem yet?
Robie Basak (racb) wrote : | #9 |
I don't think there will be a fix to this problem in Ubuntu until it is fixed upstream. I suggest you ask there.
darek334 (darek334) wrote : | #10 |
Confirmed, MySQL stops sometimes 10 minutes, there is no setting to change this ?
Kaijia Feng (fengkaijia) wrote : | #11 |
Got the same issue, found a possible explanation: my cloud provider store time in local timezone (earlier than UTC); at startup MySQL boot first, then NTP, which updates the time to UTC; therefore, MySQL literally "started in the future" (sounds interesting).
I can hotfix this by setting the timezone to UTC:
dpkg-reconfigure tzdata
After that MySQL can be stopped immediately. Even after I switch timezone back to local time, it can still stop or restart without problems. After rebooting the server (a KVM), I have to do this again unless the NTP timezone issue is fixed.
BTW, @darek334:
> MySQL stops sometimes 10 minutes
10 minutes is the timeout period defined in system/
I _think_ adding After=ntp.service would be safe (does not imply Requires), but need to learn more about the ordering and dependency system in systemd first.