Install broken, won't start

[Kim Alvefur]

After purging mysql-server, mysql-server-5.1, mysql-server-core and mysql-server-core-5.1, installing them again results in it asking for a mysql root user password 3 times, and then it gets stuck on `start msyql` which never finishes

/var/log/syslog has this, which is repeated a bunch of times

init: mysql main process (24042) terminated with status 1
init: mysql main process ended, respawning
init: mysql post-start process (24043) terminated with status 1

[Kim Alvefur (zash)]

# apt-cache policy mysql-server
mysql-server:
  Installed: (none)
  Candidate: 5.1.41-3ubuntu12.6
  Version table:
     5.1.41-3ubuntu12.6 0
        500 http://se.archive.ubuntu.com/ubuntu/ lucid-updates/main Packages
     5.1.41-3ubuntu12.3 0
        500 http://security.ubuntu.com/ubuntu/ lucid-security/main Packages
     5.1.41-3ubuntu12 0
        500 http://se.archive.ubuntu.com/ubuntu/ lucid/main Packages

[Clint Byrum (clint-fewbar)]

Zash, thanks very much for reporting this bug and working with us to make Ubuntu better!

Are you positive that you "purged" those packages, or did you just remove them? When mysql-server-5.1 is purged, it removes all of the databases, but the syslog grep shows mysql_upgrade running to bring the tables up to the latest version.

My guess is that mysqld was somehow still running from the previous installation, possibly a bug in the maintainer scripts not able to shut down mysqld properly, and so it blocked the new mysqld from starting.

Either way, I was unable to repeat this by installing 5.1.41-3ubuntu12 and then purging it, and then install 5.1.41-3ubuntu12.6. I also tried installing 5.1.41-3ubuntu12 and then upgrading.

To be clear, please provide the answers to these questions:

1. What are the exact steps you took to reach this bug (commands used would be ideal, try using the 'history' command)
2. Did you choose a mysql root password when prompted?

Marking Incomplete, please change back to 'New' upon answering those questions.

[Khairul Aizat Kamarudzzaman (fenris)]

im doing an upgrade after cloning fresh VM ... then while extracting its hang ....

[Kim Alvefur (zash)]

On Tue, 2010-08-03 at 16:59 +0000, Clint Byrum wrote:
> 1. What are the exact steps you took to reach this bug (commands used
> would be ideal, try using the 'history' command)

It broke when I upgraded from hardy to 10.04. Maybe because the machine
didn't run well with a PAE kernel and I had to do a hard reboot for it
to come back to life, or just a broken upgrade.
It used to get stuck in a xargs call in /etc/mysql/debian-start that
seemed to go forkbomb. Since then I've just been running `sudo -b -u
mysql mysqld`.
And after the recent mysql update, I tried to fix it by backing up all
the data and remove everything and reinstall, but apparently it just
broke even worse. :(

> 2. Did you choose a mysql root password when prompted?

Choosing a password results in it complaining about not being able to
change the password later, and then getting stuck on `start mysql`
again.

[Clint Byrum (clint-fewbar)]

Zash, thanks for the update. It sounds like this system was pretty seriously damaged at one time, and may need to have a fresh install. However there are a few things to check:

In the grep I notice this:

Aug 3 16:51:40 gladius mysqld[22873]: 100803 16:51:40 InnoDB: Started; log sequence number 0 0
Aug 3 16:51:40 gladius mysqld[22873]: 100803 16:51:40 [ERROR] /usr/sbin/mysqld: Can't find file: './mysql/user.frm' (errno: 13)
Aug 3 16:51:40 gladius mysqld[22873]: ERROR: 1017 Can't find file: './mysql/user.frm' (errno: 13)

errno: 13 is "permission denied". You may need to check the permissions on the /var/lib/mysql folders to make sure the directories are owned and/or writable by the 'mysql' user.

The "hang" you're experiencing appears to be mysqld restarting over and over because of this issue.

One thing to try is the following steps:

1. make sure all mysqld's are dead (stop mysql, killall mysqld, etc, verify with ps auxw | grep mysqld)
2. sudo mv /var/lib/mysql /var/lib/mysql.old ; sudo mv /etc/mysql /etc/mysql.old
3. sudo dpkg --purge mysql-server-5.1
4. sudo apt-get install mysql-server-5.1
5. stop mysql
6. mv /var/lib/mysql /var/lib/mysql.empty
7. mv /var/lib/mysql.old /var/lib/mysql
8. start mysql

If you get the same errno: 13 errors, try

chown mysql.root /var/lib/mysql/mysql
chmod 700 /var/lib/mysql/mysql
chown mysql.mysql /var/lib/mysql/mysql/*
chmod 660 /var/lib/mysql/mysql/*

Now, if mysql starts and sees your data, then you know the problem was probably a configuration issue from the /etc/mysql.old directory. However, if it still fails, then there may be some serious issues with your data. Upon failure or success, please come back and post the results, maybe another log file showing what happened.

[Clint Byrum (clint-fewbar)]

Khairul, if you could post a similar grep of your /var/log/syslog as the one above (please keep it to just the time period where you see the hang) that would be helpful to verify that yours is in fact the same issue.

[Clint Byrum (clint-fewbar)]

Setting back to Incomplete until there is a response from Zash.

[Mathias Gug (mathiaz)]

If there is a permission denied error, it may be related to mysql apparmor profile. Is /var/lib/mysql a symlink pointing to somewhere else? Looking at /var/log/kern.log should give some clue about apparmor failures.

[Kim Alvefur (zash)]

On Wed, 2010-08-04 at 15:47 +0000, Clint Byrum wrote:
> It sounds like this system was pretty
> seriously damaged at one time, and may need to have a fresh install.
Indeed :(

> chown mysql.mysql /var/lib/mysql/mysql/*
> chmod 660 /var/lib/mysql/mysql/*
Those files are owned by root:root and chmod 660 after a fresh (purge,
no /var/lib/mysql, and) install.

I don't have the original database files anymore, so I can't put those
back.

> Now, if mysql starts and sees your data, then you know the problem was
> probably a configuration issue from the /etc/mysql.old directory.
> However, if it still fails, then there may be some serious issues with
> your data. Upon failure or success, please come back and post the
> results, maybe another log file showing what happened.

`start mysql` gives
start: Job failed to start

and this in syslog:
mysql pre-start process (3541) terminated with status 1

However, `sudo -u mysql mysqld_safe` worked, and I could connect with
the root account and no password. I did set a password during install,
and it later complained about failing to change it.

[Kim Alvefur (zash)]

On Wed, 2010-08-04 at 16:45 +0000, Mathias Gug wrote:
> If there is a permission denied error, it may be related to mysql
> apparmor profile.
I tried stopping apparmor, no difference.

> Is /var/lib/mysql a symlink pointing to somewhere
> else?
No

[Kim Alvefur (zash)]

Currently running from `sudo -b -H -u mysql mysqld_safe`.

[Clint Byrum (clint-fewbar)]

Zash, on reading your 'output of apt-get install and ls" text attachment, it contradicts what you stated in comment #10, which was

>> chown mysql.mysql /var/lib/mysql/mysql/*
>> chmod 660 /var/lib/mysql/mysql/*
>Those files are owned by root:root and chmod 660 after a fresh (purge,
>no /var/lib/mysql, and) install.

The ls command output clearly shows the files owned as expected, by mysql/root or myslq/mysql.

Given the fact that those permissions should allow the mysql user to access and read those directories fully, I'm wondering if there aren't some fundamental problems with your system. Can you share the output of these commands:

ls -d /var/lib
ls -d /var
getent passwd | grep mysql

Also is it possible you have a locally customized file in /etc/init/mysql.conf or /etc/init.d/mysqld ?

Thanks!

[Kim Alvefur (zash)]

On Mon, 2010-08-09 at 05:09 +0000, Clint Byrum wrote:
> Zash, on reading your 'output of apt-get install and ls" text
> attachment, it contradicts what you stated in comment #10, which was
>> [...]
> The ls command output clearly shows the files owned as expected, by
> mysql/root or myslq/mysql.

It was /var/lib/mysql/mysql/* which was root:root and the ls output is
the parent dir.

> Can you share the output of these commands:
>
> ls -d /var/lib
> ls -d /var
just outputs /var/lib and /var

> getent passwd | grep mysql
~$ getent passwd | grep mysql
mysql:x:110:117:MySQL Server,,,:/var/lib/mysql:/bin/false

> Also is it possible you have a locally customized file in
> /etc/init/mysql.conf or /etc/init.d/mysqld ?
Nope

[Clint Byrum (clint-fewbar)]

Kim, I'm sorry this bug has gone unanswered for so long. Do you still have this problem? I meant

ls -ld

not

ls -d

Anyway, marking Incomplete again. Please let me know if you are still affected.

[Launchpad Janitor (janitor)]

[Expired for mysql-dfsg-5.1 (Ubuntu) because there has been no activity for 60 days.]