startup race between MariaDB and nova services

[Andreas Friedrich]

I've followed the instructions from http://docs.openstack.org/juno/install-guide/install/apt/content/ and build an OpenStack system with one controller, one network and one compute node. All three nodes are virtual machines. When starting up the three machines the services nova-cert and nova-scheduler on the controller node are always down:

nova service-list
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| 1 | nova-cert | controller | internal | enabled | down | 2015-03-16T08:20:47.000000 | - |
| 2 | nova-consoleauth | controller | internal | enabled | up | 2015-03-16T08:30:01.000000 | - |
| 3 | nova-scheduler | controller | internal | enabled | down | 2015-03-16T08:20:51.000000 | - |
| 4 | nova-conductor | controller | internal | enabled | up | 2015-03-16T08:29:57.000000 | - |
| 5 | nova-compute | compute1 | nova | enabled | up | 2015-03-16T08:30:01.000000 | - |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+

It is the same problem as described in https://ask.openstack.org/en/question/60948/nova-cert-and-nova-scheduler-not-started-after-reboot/.

I think the problem is, that MariaDB is started via /etc/init/S19mysql while the nova services are started via upstart. When the nova services are starting, mysql was not ready:

/var/log/nova/nova-cert.log:
2015-03-16 09:21:36.253 930 TRACE nova.openstack.common.threadgroup OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on 'controller' (111)") None None

/var/log/nova/nova-scheduler.log:
2015-03-16 09:21:36.251 935 TRACE nova.openstack.common.threadgroup OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on 'controller' (111)") None None

/var/log/mysql/error.log:
150316 9:21:39 [Note] /usr/sbin/mysqld: ready for connections.

If you look at the time stamps, mysql reported "ready" after the nova services had logged an error. Maybe that's the problem.

[Stefano Maffulli (smaffulli)]

Try to use the support channels listed on https://www.openstack.org/community#help