mysql server crashes with port 3306 already in use

[Raghvendra Khatri]

[Impact]

When a user attepts to install mysql-server to their system, there is a chance that it will fail due to a race condition in the post-install script.
In this situation, the user will be provided with the error:

dpkg: error processing package mysql-server-8.0 (--configure):
 installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1

The only way to fix this is to uninstall then retry the install again.

The provided fix resolves the race condition by making sure mysql is shut down completely after setup before it boots back up again for use by the user.

[Test Plan]

Since this is a race condition, it can take a while to reproduce. However, using less cores and lower CPU speeds makes it appear more often. Testing with the following code many times will eventually show that, prior to the update, this issue appears. Meanwhile after the update it does not.

# lxc launch images:ubuntu/[kinetic, jammy, impish, focal] test-mysql
# lxc exec test-mysql bash
# apt update && apt dist-upgrade -y
# apt install mysql-server

[Where problems could occur]

Since this update has the postinst script enforce shutdown after configuration, mysql may be cut off abruptly, which could cause internal problems that it may need to address on the following startup.

[Other Info]

The fix was released to kinetic in 8.0.29-0ubuntu4

[Original Description]

mysql server not running

ProblemType: Package
DistroRelease: Ubuntu 20.04
Package: mysql-server-8.0 8.0.23-0ubuntu0.20.04.1
ProcVersionSignature: Ubuntu 5.8.0-48.54~20.04.1-generic 5.8.18
Uname: Linux 5.8.0-48-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.16
Architecture: amd64
CasperMD5CheckResult: skip
Date: Thu Mar 25 15:33:48 2021
ErrorMessage: installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1
InstallationDate: Installed on 2021-03-24 (0 days ago)
InstallationMedia: Ubuntu 20.04.2.0 LTS "Focal Fossa" - Release amd64 (20210209.1)
Logs.var.log.daemon.log:

MySQLConf.etc.mysql.conf.d.mysql.cnf: [mysql]
MySQLConf.etc.mysql.conf.d.mysqldump.cnf:
 [mysqldump]
 quick
 quote-names
 max_allowed_packet = 16M
MySQLVarLibDirListing: ['#innodb_temp', 'ca-key.pem', 'binlog.000008', 'client-key.pem', 'server-key.pem', 'sys', 'binlog.000007', 'ibtmp1', 'server-cert.pem', 'binlog.000004', 'client-cert.pem', 'undo_002', 'binlog.000013', 'mysql', 'performance_schema', 'ib_logfile0', 'auto.cnf', 'binlog.000005', 'binlog.000002', 'ib_logfile1', 'debian-5.7.flag', 'binlog.000001', 'private_key.pem', 'binlog.000015', 'binlog.000011', 'ibdata1', 'ca.pem', 'binlog.000009', 'binlog.index', 'public_key.pem', 'binlog.000003', 'ib_buffer_pool', 'binlog.000014', 'binlog.000016', '#ib_16384_0.dblwr', 'binlog.000006', 'mysql.ibd', '#ib_16384_1.dblwr', 'binlog.000012', 'raghvendra-HP-EliteBook-Folio-9470m.pid', 'undo_001', 'binlog.000010']
ProcCmdline: BOOT_IMAGE=/boot/vmlinuz-5.8.0-48-generic root=UUID=7d0f25d1-2e3e-4c9a-aabc-c337c6b1278b ro quiet splash vt.handoff=7
Python3Details: /usr/bin/python3.8, Python 3.8.5, python3-minimal, 3.8.2-0ubuntu2
PythonDetails: N/A
RelatedPackageVersions:
 dpkg 1.19.7ubuntu3
 apt 2.0.4
SourcePackage: mysql-8.0
Title: package mysql-server-8.0 8.0.23-0ubuntu0.20.04.1 failed to install/upgrade: installed mysql-server-8.0 package post-installation script subprocess returned error exit status 1
UpgradeStatus: No upgrade log present (probably fresh install)

[Utkarsh Gupta (utkarsh)]

Hello Raghvendra,

Thanks for taking out time to file a bug and making Ubuntu server better.

This bug is a duplicate of LP: #1899248 (cf: https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1899248) and thus I've marked the same. This is being worked out and people are aware of this already.

In case you think this isn't a duplicate or should you need any further assistance, please don't hesitate to reach out. Thanks, again.

[Lena Voytek (lvoytek)]

Hi,

After looking further into bug reports marked as duplicates of LP: #1899248, I noticed a separate error that is common between some of them. Although this is marked as a duplicate due to the error:

[ERROR] [MY-011300] [Server] Plugin mysqlx reported: 'Setup of socket: '/var/run/mysqld/mysqlx.sock' failed, can't create lock file /var/run/mysqld/mysqlx.sock.lock'

the report was more likely generated due to the following in the mysql error.log:

2021-03-25T10:57:58.560178Z 0 [ERROR] [MY-010262] [Server] Can't start server: Bind on TCP/IP port: Address already in use
2021-03-25T10:57:58.560457Z 0 [ERROR] [MY-010257] [Server] Do you already have another mysqld server running on port: 3306 ?

Therefore I will remove the duplicate status of this bug and use it for all bugs related to this specific issue.

Thanks

[Utkarsh Gupta (utkarsh)]

Hey, is there any update on this, Lena? :D

[Lena Voytek (lvoytek)]

Just started work on this again last week. I'm still working on reproducing the error consistently

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-8.0 - 8.0.29-0ubuntu4

---------------
mysql-8.0 (8.0.29-0ubuntu4) kinetic; urgency=medium

  * d/mysql-server-8.0.postinst: Confirm mysqld shuts down with stop_server after
    initialization to avoid overlapping use of port 3306 (LP: #1921378)

 -- Lena Voytek <email address hidden> Wed, 15 Jun 2022 14:27:47 -0700

[Steve Langasek (vorlon)]

Hello Raghvendra, or anyone else affected,

Accepted mysql-8.0 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-8.0/8.0.29-0ubuntu0.22.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted mysql-8.0 (8.0.29-0ubuntu0.22.04.3) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

exim4/4.95-4ubuntu2.1 (ppc64el)
ruby-mysql2/0.5.3-3ubuntu4 (armhf, amd64, s390x, arm64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#mysql-8.0

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Robie Basak (racb)]

Impish will EOL in three days so there's no point SRUing this fix to Impish as the process would take longer than that.

[Lena Voytek (lvoytek)]

Verified through many iterations of:

# lxc launch images:ubuntu/jammy test-mysql
# lxc exec test-mysql bash
# apt update && apt dist-upgrade -y
# apt install mysql-server

and

# lxc launch images:ubuntu/jammy test-mysql
# lxc exec test-mysql bash
# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF
# apt update && apt dist-upgrade -y
# apt install mysql-server

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-8.0 - 8.0.29-0ubuntu0.22.04.3

---------------
mysql-8.0 (8.0.29-0ubuntu0.22.04.3) jammy; urgency=medium

  * d/a/source_mysql-8.0.py: Fix apport too many symbolic
    links report for my.cnf (LP: #1969369)
  * d/mysql-server-8.0.postinst: Confirm mysqld shuts down with stop_server
    after initialization to avoid overlapping use of port 3306 (LP: #1921378)

 -- Lena Voytek <email address hidden> Mon, 27 Jun 2022 10:55:12 -0700

[Robie Basak (racb)]

The verification of the Stable Release Update for mysql-8.0 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Steve Langasek (vorlon)]

An upload of mysql-8.0 to focal-proposed has been rejected from the upload queue for the following reason: "older thannnnn version currently in focal-updates".

[Chris Halse Rogers (raof)]

Hello Raghvendra, or anyone else affected,

Accepted mysql-8.0 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-8.0/8.0.30-0ubuntu0.20.04.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lena Voytek (lvoytek)]

Verified through many iterations of:

# lxc launch images:ubuntu/focal test-mysql
# lxc exec test-mysql bash
# apt update && apt dist-upgrade -y
# apt install mysql-server

and

# lxc launch images:ubuntu/focal test-mysql
# lxc exec test-mysql bash
# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF
# apt update && apt dist-upgrade -y
# apt install mysql-server

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted mysql-8.0 (8.0.30-0ubuntu0.20.04.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

postfix/3.4.13-0ubuntu1.2 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#mysql-8.0

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Robie Basak (racb)]

Hello Raghvendra, or anyone else affected,

Accepted mysql-8.0 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/mysql-8.0/8.0.31-0ubuntu0.20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Lena Voytek (lvoytek)]

Reverified through many iterations of:

# lxc launch images:ubuntu/focal test-mysql
# lxc exec test-mysql bash
# apt update && apt dist-upgrade -y
# apt install mysql-server

and

# lxc launch images:ubuntu/focal test-mysql
# lxc exec test-mysql bash
# cat <<EOF >/etc/apt/sources.list.d/ubuntu-$(lsb_release -cs)-proposed.list
# Enable Ubuntu proposed archive
deb http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs)-proposed restricted main multiverse universe
EOF
# apt update && apt dist-upgrade -y
# apt install mysql-server

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted mysql-8.0 (8.0.31-0ubuntu0.20.04.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

lighttpd/1.4.55-1ubuntu1.20.04.1 (armhf)
kopanocore/8.7.0-7ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#mysql-8.0

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Lena Voytek (lvoytek)]

Re-ran the regressed autopkgtests yesterday, both seem to be succeeding now and were just flaky

[Launchpad Janitor (janitor)]

This bug was fixed in the package mysql-8.0 - 8.0.31-0ubuntu0.20.04.2

---------------
mysql-8.0 (8.0.31-0ubuntu0.20.04.2) focal; urgency=medium

  * d/mysql-server-8.0.postinst: Confirm mysqld shuts down with stop_server
    after initialization to avoid overlapping use of port 3306 (LP: #1921378)

 -- Lena Voytek <email address hidden> Tue, 15 Nov 2022 13:42:17 -0700