nova-conductor starts before database tables are created

[Ryan Moe]

mirantis: "yes"
  production: "docker"
  release: "5.0.1"
  api: "1.0"
  build_number: "169"
  build_id: "2014-08-11_12-45-06"
  astute_sha: "6db5f5031b74e67b92fcac1f7998eaa296d68025"
  fuellib_sha: "a31dbac8fff9cf6bc4cd0d23459670e34b27a9ab"
  ostf_sha: "09b6bccf7d476771ac859bb3c76c9ebec9da9e1f"
  nailgun_sha: "04ada3cd7ef14f6741a05fd5d6690260f9198095"
  fuelmain_sha: "43374c706b4fdce28aeb4ef11e69a53f41646740"

nova-conductor started 24 minutes prior to nova's database tables being created. This caused the conductor service to fail to start and constantly spawn new children. During the deployment nova-conductor CPU usage was between 20% and 80%.

The log is full of errors like this:

2014-08-13 00:03:14.850 28556 TRACE nova.openstack.common.threadgroup ProgrammingError: (ProgrammingError) (1146, "Table 'nova.services' doesn't exist") 'SELECT services.cre
ated_at AS services_created_at, services.updated_at AS services_updated_at, services.deleted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS
services_id, services.host AS services_host, services.`binary` AS services_binary, services.topic AS services_topic, services.report_count AS services_report_count, services
.disabled AS services_disabled, services.disabled_reason AS services_disabled_reason \nFROM services \nWHERE services.deleted = %s AND services.host = %s AND services.`binar
y` = %s \n LIMIT %s' (0, 'node-6', 'nova-conductor', 1)
2014-08-13 00:03:14.850 28556 TRACE nova.openstack.common.threadgroup
2014-08-13 00:03:14.936 7741 INFO nova.openstack.common.service [-] Child 28556 exited with status 0
2014-08-13 00:03:14.940 7741 INFO nova.openstack.common.service [-] Started child 28880

These messages started at 23:39 and the database tables weren't created until 00:03.

[Ryan Moe (rmoe)]

I've confirmed this on 5.0.1 and it might still be an issue with 5.1. That still needs to be tested.

[Dmitry Mescheryakov (dmitrymex)]

At first sight, this seems to be purely problem of puppet manifests, hence removing MOS from affected projects. Talked with Sergey Golovatiuk about it. He will contact MOS Components team if our assistance is required.

[Mike Scherbakov (mihgen)]

Sergii, are you working on this bug? Or we could reassign it back to the team, so someone else can take it?

[Mike Scherbakov (mihgen)]

Also, is it a Medium priority truly? If it is not reproduced often and env stabilizes in 20 min (nova-conductor works fine after 20min), I think it can be moved to 6.0.

[Sergii Golovatiuk (sgolovatiuk)]

It's medium priority as it causes High load without "Service disruption". It can be moved to 6.0 but patches should be backported to all supported versions.

[Dmitry Borodaenko (angdraug)]

Won't Fix for 5.1.x: priority is not High or Critical.

[Stanislav Makar (smakar)]

I haven't come across such error on my env (Fuel 6.0, Juno on Centos 6.5, HA VLAN, ISO: fuel-6.0-79-2014-11-05_21-28-15.iso )

[Aleksandr Didenko (adidenko)]

It looks like it's not a problem anymore after "nova" module upgrade:

Fri Nov 07 11:30:16 +0000 2014 Puppet (debug): Adding relationship from Exec[nova-db-sync] to Service[nova-conductor] with 'notify'
Fri Nov 07 11:30:26 +0000 2014 /Stage[main]/Nova::Api/Exec[nova-db-sync]/notify (debug): subscribes to Service[nova-conductor]
Fri Nov 07 11:42:13 +0000 2014 /Stage[main]/Nova::Api/Exec[nova-db-sync] (info): Scheduling refresh of Service[nova-conductor]

As you can see, Exec[nova-db-sync] notifies Service[nova-conductor], which means db-sync will be evaluated before Service[nova-conductor]

[Aleksandr Didenko (adidenko)]

5.1 should not be affected as well, since we've upgraded "nova" module in 5.1.