Nova services fail to start with DB error: "DBConnectionError: (OperationalError) (1047, 'Unknown command') None None"

[Tatyanka]

OS: Ubuntu
 Scenario:
            1. Create cluster
            2. Add 3 nodes with controller roles
            3. Add 2 nodes with compute roles
            4. Deploy the cluster
            5. Validate cluster was set up correctly, there are no dead
            services, there are no errors in logs

Actual Result:
Nova scheduler and conductor were not started on one of the node with next error:
http://paste.openstack.org/show/220828/

At the same time I've started it manually without errors

[root@nailgun ~]# cat /etc/fuel/version.yaml
VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "6.1"
  openstack_version: "2014.2.2-6.1"
  api: "1.0"
  build_number: "395"
  build_id: "2015-05-08_11-08-49"
  nailgun_sha: "46f55c293e4540d31bcaa6ca3fba77235fb27537"
  python-fuelclient_sha: "af6c9c3799b9ec107bcdc6dbf035cafc034526ce"
  astute_sha: "6a4dcd11c67af2917815f3678fb594c7412a4c97"
  fuel-library_sha: "f385d6a58298c702f8d4f14c452dcffdc0b1e2a3"
  fuel-ostf_sha: "740ded337bb2a8a9b3d505026652512257375c01"
  fuelmain_sha: "3eca5e8f7ca6a83faff5feeca92c21cff31c0af1"
[root@nailgun ~]#

[Pavel Kholkin (pkholkin)]

I don't think it is a nova problem. According to logs, mysql was down at that time:
http://paste.openstack.org/show/227140/

Please look at the file 10.109.20.2/var/log/docker-logs/remote/node-1.test.domain.local/haproxy.log in attached fuel-snapshot.
Thanks!

[Roman Podoliaka (rpodolyaka)]

Pavel, we still want nova-* services to survive MySQL outage and continue to work, when it's back online.

Could you please give the next scenario a try?

1) deploy an env
2) turn off mysql (or just block it by the means of iptables)
3) restart nova-scheduler, nova-conductor, etc
4) make sure nova-* services try to connect to the DB and fail
5) turn on mysql again
6) make sure nova-scheduler and nova-conductor operate normally

[Pavel Kholkin (pkholkin)]

Roman, we tried this scenario, services were stopped after loosing connection to mysql, we will add 'respawn' to /etc/init/nova-*.conf to restart dead services .

[Daniil Trishkin (dtrishkin)]

[0] - This commit fixed that issue, at that moment Thomas has rebuilt openstack-pkg-tools [1] with respawn addition, so package was rebuilded with this addition to upstart scripts
[0] - https://review.fuel-infra.org/#/c/8470/
[1] - https://review.fuel-infra.org/#/c/7581/