package mysql-server-8.0 8.0.33-0ubuntu0.22.04.4 failed to install/upgrade: installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1 on upgrade from Fossa after an upgrade from Bionic

[FGM]

This happened on a Bionic server install. I upgraded to Focal, using the maintainer config file instead of mine, with no apparent issue.

I then upgraded to Jammy and this happened near the end of the upgrade.

ProblemType: Package
DistroRelease: Ubuntu 22.04
Package: mysql-server-8.0 8.0.33-0ubuntu0.22.04.4
ProcVersionSignature: Ubuntu 5.4.0-155.172-generic 5.4.235
Uname: Linux 5.4.0-155-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.5
Architecture: amd64
CasperMD5CheckResult: unknown
CloudArchitecture: x86_64
CloudID: aws
CloudName: aws
CloudPlatform: ec2
CloudRegion: eu-west-3
CloudSubPlatform: metadata (http://169.254.169.254)
Date: Tue Aug 1 14:19:57 2023
Ec2AMI: ami-04feb8577746fe710
Ec2AMIManifest: (unknown)
Ec2Architecture: x86_64
Ec2AvailabilityZone: eu-west-3a
Ec2Imageid: ami-04feb8577746fe710
Ec2InstanceType: t3.medium
Ec2Instancetype: t3.medium
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
Ec2Region: eu-west-3
ErrorMessage: installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1
Logs.var.log.daemon.log:

MySQLConf.etc.mysql.conf.d.mysql.cnf: [mysql]
MySQLConf.etc.mysql.conf.d.mysqld_safe_syslog.cnf:
 [mysqld_safe]
 syslog
MySQLConf.etc.mysql.conf.d.mysqldump.cnf:
 [mysqldump]
 quick
 quote-names
 max_allowed_packet = 16M
MySQLConf.etc.mysql.my.cnf: my.cnf links to /etc/mysql/mysql.cnf
MySQLConf.etc.mysql.mysql.conf.d.mysqld_safe_syslog.cnf:
 [mysqld_safe]
 syslog
MySQLVarLibDirListing: ['auto.cnf', 'debian-5.7.flag', 'mysql', 'binlog.index']
ProcCmdline: BOOT_IMAGE=/boot/vmlinuz-5.4.0-155-generic root=UUID=8d4e0b32-1ae3-419c-8df3-44e97cd83b75 ro console=tty1 console=ttyS0
Python3Details: /usr/bin/python3.10, Python 3.10.12, python3-minimal, 3.10.6-1~22.04
PythonDetails: N/A
RebootRequiredPkgs: Error: path contained symlinks.
RelatedPackageVersions:
 dpkg 1.21.1ubuntu2.2
 apt 2.4.9
SourcePackage: mysql-8.0
Title: package mysql-server-8.0 8.0.33-0ubuntu0.22.04.4 failed to install/upgrade: installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1
UpgradeStatus: Upgraded to jammy on 2023-08-01 (0 days ago)

[Athos Ribeiro (athos-ribeiro)]

These are the relevant snippets from the attached logs:

Setting up mysql-server-8.0 (8.0.33-0ubuntu0.22.04.4) ...
Specified filename /var/lib/mysql/ibdata1 does not exist.
...
[ERROR] [MY-010946] [Server] Failed to start mysqld daemon. Check mysqld error log.

And in the mysql log, we see:

[System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.20.04.4) starting as process 1569
[ERROR] [MY-011011] [Server] Failed to find valid data directory.
[ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
[ERROR] [MY-010119] [Server] Aborting
[System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.20.04.4) (Ubuntu).
[System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33-0ubuntu0.22.04.4) starting as process 65221
[ERROR] [MY-011011] [Server] Failed to find valid data directory.
[ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
[ERROR] [MY-010119] [Server] Aborting
[System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.04.4) (Ubuntu).

which shows that the service was already failing to start for the 20.04 package.

Finally, the old config file has an entry:

datadir = /var/lib/jenkins/mysql

while the new config file has no such entry.

From the data above, I suppose your initial mysql installation used /var/lib/jenkins/mysql and mysql was never set to run using the default datadir.

Since updating the data in the datadir is part of the mysql upgrade process, you should not accept replacing your configuration files with the default ones during the package upgrade process, since you do have a custom mysql setup.

To upgrade the package, you should instead ensure your configuration file is compatible with the new mysql version.

Since this seems to be a configuration issue rather than a bug in the package, I am marking this bug as incomplete for now. If you disagree with the assessment, please state the reasons you believe this is indeed a bug with the mysql package and set the status of this bug back to new.

[Athos Ribeiro (athos-ribeiro)]

After discussing this specific bug within the server team, we decided to push it to our backlog for further investigation on whether is there is anything we can do to improve the user experience for this sort of upgrades.

FGM, would you mind trying to propserly set your configuration file and re-trying the operation, and then letting us know how it goes for you?

[FGM (fgm)]

After that error, I reapplied the various specific config tweaks I had applied in the previous config on 18.04, including the dataDir, and my DBs have been working normally indeed.

[Paride Legovini (paride)]

Hello FGM, that's good to hear. I'm leaving this bug as Triaged to track "bad UX" issue, as Athos mentioned.

[Qingshen Peng (pengq01)]

I have almost exactly the same error when I upgrade a Ubuntu 18 server to Ubuntu 20.

root@scorm-test01:~# apt-get install mysql-server
Do you want to continue? [Y/n] y
Specified filename /var/lib/mysql/ibdata1 does not exist.
mysqld will log errors to /var/log/mysql/error.log
2023-11-30T19:27:46.141216Z 0 [ERROR] [MY-010946] [Server] Failed to start mysqld daemon. Check mysqld error log.
Warning: Unable to start the server.

root@scorm-test01:~# more /var/log/mysql/error.log
...
2023-12-01T00:44:06.365705Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.35-0ubuntu0.20.04.1) starting as process 12266
2023-12-01T00:44:06.379451Z 1 [ERROR] [MY-011011] [Server] Failed to find valid data directory.
2023-12-01T00:44:06.379550Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2023-12-01T00:44:06.379572Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-12-01T00:44:06.381283Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.35-0ubuntu0.20.04.1) (Ubuntu).
root@scorm-test01:~#

root@scorm-test01:~# more /var/log/mysql/error.log
2023-12-01T00:44:06.365705Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.35-0ubuntu0.20.04.1) starting as process 12266
2023-12-01T00:44:06.379451Z 1 [ERROR] [MY-011011] [Server] Failed to find valid data directory.
2023-12-01T00:44:06.379550Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2023-12-01T00:44:06.379572Z 0 [ERROR] [MY-010119] [Server] Aborting
2023-12-01T00:44:06.381283Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.35-0ubuntu0.20.04.1) (Ubuntu).

2023-12-01T00:44:06.381283Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.35-0ubuntu0.20.04.1) (Ubuntu).
2023-12-01T01:06:16.808982Z 0 [Warning] [MY-010091] [Server] Can't create test file /data/mysqld_tmp_file_case_insensitive_test.lower-test
2023-12-01T01:06:16.809055Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.35-0ubuntu0.20.04.1) starting as process 14880
2023-12-01T01:06:16.811554Z 0 [Warning] [MY-010091] [Server] Can't create test file /data/mysqld_tmp_file_case_insensitive_test.lower-test
2023-12-01T01:06:16.811563Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /data/ is case insensitive
mysqld: File './binlog.index' not found (OS errno 13 - Permission denied)

root@scorm-test01:~# tail -20 /var/log/mysql/error.log
...
2023-12-01T01:38:33.628675Z 0 [Warning] [MY-010091] [Server] Can't create test file /data/mysql/mysqld_tmp_file_case_insensitive_test.lower-test
2023-12-01T01:38:33.628684Z 0 [Warning] [MY-010159] [Server] Setting lower_case_table_names=2 because file system for /data/mysql/ is case insensitive
mysqld: File './binlog.index' not found (OS errno 13 - Permission denied)

An Ubuntu Support Case 00374645 created for three days, with priority level 2, but as of today, we have not received any response from the Ubuntu support team.
The OS version upgrade script
do-release-upgrade
doesn't upgrade mysql on the server correctly. After the upgrade, mysql stopped starting up.