mysql service holds deleted log files
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
MySQL InnoDB Cluster Charm |
Triaged
|
High
|
Unassigned |
Bug Description
I've observed that two very large files were held by mysql process causing the disk to be 95% utilized.
ubuntu@
Last login: Thu Feb 1 17:30:08 2024 from 10.131.8.4
ubuntu@
lsof: WARNING: can't stat() tracefs file system /sys/kernel/
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
unattende 332 root 3w REG 8,34 113 0 301747 /var/log/
mysqld 348 mysql 1w REG 8,34 104091260620 0 1750078 /var/log/
mysqld 348 mysql 2w REG 8,34 104091260620 0 1750078 /var/log/
mysqld 348 mysql 6u REG 8,34 0 0 134193 /tmp/#134193 (deleted)
mysqld 348 mysql 7u REG 8,34 142 0 134226 /tmp/#134226 (deleted)
mysqld 348 mysql 8u REG 8,34 64710 0 136080 /tmp/#136080 (deleted)
mysqld 348 mysql 13u REG 8,34 2174 0 136916 /tmp/#136916 (deleted)
landscape 1118 root 6w REG 8,34 361 0 304942 /var/log/
landscape 1120 landscape 6w REG 8,34 2207957 0 305187 /var/log/
landscape 1121 landscape 6w REG 8,34 1690223 0 305186 /var/log/
landscape 1122 root 6w REG 8,34 536214 0 305185 /var/log/
Two /var/log/
After restarting mysql service, the files got released and disk utilization went down.
ubuntu@
ubuntu@
lsof: WARNING: can't stat() tracefs file system /sys/kernel/
Output information may be incomplete.
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
unattende 332 root 3w REG 8,34 113 0 301747 /var/log/
landscape 1118 root 6w REG 8,34 361 0 304942 /var/log/
landscape 1120 landscape 6w REG 8,34 2207957 0 305187 /var/log/
landscape 1121 landscape 6w REG 8,34 1690223 0 305186 /var/log/
landscape 1122 root 6w REG 8,34 536214 0 305185 /var/log/
mysqld 1773156 mysql 6u REG 8,34 0 0 134193 /tmp/#134193 (deleted)
mysqld 1773156 mysql 7u REG 8,34 0 0 134226 /tmp/#134226 (deleted)
mysqld 1773156 mysql 8u REG 8,34 0 0 136080 /tmp/#136080 (deleted)
mysqld 1773156 mysql 13u REG 8,34 0 0 136916 /tmp/#136916 (deleted)
ubuntu@
Filesystem Size Used Avail Use% Mounted on
/dev/sdc2 229G 108G 109G 50% /
none 492K 4.0K 488K 1% /dev
tmpfs 100K 0 100K 0% /dev/lxd
tmpfs 100K 0 100K 0% /dev/.lxd-mounts
tmpfs 189G 0 189G 0% /dev/shm
tmpfs 76G 328K 76G 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
snapfuse 56M 56M 0 100% /snap/core18/2812
snapfuse 75M 75M 0 100% /snap/core22/1033
snapfuse 64M 64M 0 100% /snap/core20/1891
snapfuse 112M 112M 0 100% /snap/lxd/24322
snapfuse 94M 94M 0 100% /snap/mysql-
snapfuse 6.2M 6.2M 0 100% /snap/prometheu
snapfuse 41M 41M 0 100% /snap/snapd/20290
snapfuse 64M 64M 0 100% /snap/core20/2105
snapfuse 41M 41M 0 100% /snap/snapd/20671
snapfuse 115M 115M 0 100% /snap/lxd/26741
snapfuse 270M 270M 0 100% /snap/grafana-
tmpfs 38G 4.0K 38G 1% /run/user/1000
Ubuntu 22.04.3 LTS
Openstack Yoga
Juju 3.1.6
LXD 5.0.2
MAAS 3.2.10
How to reproduce: monitor disk utilization and the output of "sudo lsof +L1"
Noticed this error in the journalctl output: "Feb 01 20:17:01 juju-c3cb23-0-lxd-2 sudo[356539]: root : command not allowed ; PWD=/ ; USER=root ; COMMAND=
The error was repeated many times.
Changed in charm-mysql-innodb-cluster: | |
status: | Expired → Triaged |
importance: | Undecided → High |
Found this in journalctl output:
Feb 02 00:00:09 juju-c3cb23-7-lxd-8 logrotate[1754765]: error: error running shared postrotate script for '/var/log/mysql.log /var/log/mysql/*log '
Feb 02 00:00:09 juju-c3cb23-7-lxd-8 systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Feb 02 00:00:09 juju-c3cb23-7-lxd-8 systemd[1]: logrotate.service: Failed with result 'exit-code'.
Feb 02 00:00:09 juju-c3cb23-7-lxd-8 systemd[1]: Failed to start Rotate log files.
When I try to execute mysql-server job (/etc/logrotate .d/mysql- server) , I get this error:
root@juju- c3cb23- 7-lxd-8: /etc/logrotate. d# MYADMIN= "/usr/bin/ mysqladmin --defaults- file=/etc/ mysql/debian. cnf"
root@juju- c3cb23- 7-lxd-8: /etc/logrotate. d# $MYADMIN ping
mysqladmin: connect to server at 'localhost' failed