mysql upgrades fail when mysql data is on block storage
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| mysql (Juju Charms Collection) |
New
|
Undecided
|
Unassigned | ||
Bug Description
What happened: Landscape upgraded mysql.
What I expected: mysql would be upgraded and service would remain available
What happened instead: mysql failed during the mysql_upgrade process and the service was left offline and required manual intervention to restore.
Cause: Existence of lost+found directory in block storage root of mysql data directory causes mysql_upgrade process to fail.
Explanation: lost+found directory is removed by the charm during the upgrade to block storage, but fsck on that filesystem will recreate the directory, causing subsequent mysql_upgrade's to fail.
Proposed solution: ensure lost+found filesystem is detected and removed prior to the mysql_upgrade process.
Additional detail:
On a trusty host, during a landscape mysql update, mysql failed and the machine was left without MySQL answering on TCP ports or local sockets.
The issue is due to the lost+found directory being present on the block storage mount upon mysql_upgrade. The mysql_upgrade process mistakes the lost+found for a database (which is obviously incorrect) and blows up, leaving the database non-functional.
The issue is a known mysql issue, which has a workaround (ignore_data_dir in my.conf) in a later version of MySQL (5.6.3) but trusty currently has 5.5.55. (the bug is here: https:/
The Charm takes this into account by removing lost+found during the migration to block storage ( http://
Unfortunately, when fsck runs on the mysql data storage filesystem, the lost+found directory will be recreated. Once this happens, the next mysql_upgrade will crash and leave the server unresponsive.
