Retry of neutron-db-sync doesn't work if execution fails during tables creation

[Alexander Rubtsov]

Release: MOS 9.2

The corresponding excerpts from Puppet log file:
/Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns (debug): Exec try 1/10
Exec[neutron-db-sync](provider=posix) (debug): Executing 'neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head'
Puppet (debug): Executing 'neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head'
/Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns (debug): Sleeping for 5.0 seconds between tries
.....
/Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns (debug): Exec try 10/10
Exec[neutron-db-sync](provider=posix) (debug): Executing 'neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head'
Puppet (debug): Executing 'neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head'
/Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns (debug): Sleeping for 5.0 seconds between tries
/Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns (debug): Sleeping for 5.0 seconds between tries
.....
/Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync]/returns (notice): sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1050, "Table 'agents' already exists") [SQL: u"\nCREATE TABLE agents (\n\tid VARCHAR(36) NOT NULL, \n\tagent_type VARCHAR(255) NOT NULL,.....]
/Stage[main]/Neutron::Db::Sync/Exec[neutron-db-sync] (err): Failed to call refresh: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head returned 1 instead of one of [0]
...
http://paste.openstack.org/show/x8rHSl9ErSvD9tivzj4h/

This failure doesn't make Fuel mark the entire deployment as failed, which is wrong because actually Neutron is unable to operate.

This issue is rarely reproducible.
It seems that it occurs only if creation/population of MySQL tables was interrupted in the middle of the process.

[Alexander Rubtsov (arubtsov)]

sla2 for 9.0-updates

[Oleksiy Molchanov (omolchanov)]

Alexander,

1) Is this initial fuel deploy?
2) Can we have full diagnostic snapshot?

[Alexander Rubtsov (arubtsov)]

Oleksiy,

1) I will ask the customer about that
2) Unfortunately, the log files from the problematic deployment are not available anymore.

[Alexander Rubtsov (arubtsov)]

The customer was not able to reproduce the issue again to collect the diagnostic snapshot.

[Alexander Rubtsov (arubtsov)]

The issue has appeared again and the customer was able to collect the diagnostic snapshot this time. Please contact me so that I will provide you with the log files directly.

[Alexander Rubtsov (arubtsov)]

sla1 for 9.0-updates

[Alexander Rubtsov (arubtsov)]

It seems the issue is wider than just neutron-db-sync. The same customer has hit similar incident with Cinder database.

[Denis Meltsaykin (dmeltsaykin)]

Looks like this is a rarely occurring issue which is not that easy to fix, see also https://bugs.launchpad.net/fuel/+bug/1641136.

[Denis Meltsaykin (dmeltsaykin)]

Ok, so it seems that the root cause of the issue is some kind of high CPU load during deployment which caused MySQL to be broken. This leads to failures of the db syncs which are mostly executed as a "refreshonly = true" events and the well known bug in Puppet lets the flow to continue even if refreshonly events are failed. But it shouldn't be a big problem since while you have non-working MySQL the deployment will still be failed. Regarding the possible fixes: db-syncs are known to be non-idempotent in older versions, so in Mitaka it's just dangerous to run *-db-sync over an already prepared database so these events *must be* "refreshonly". And on the other hand we cannot change the puppet code itself or update puppet to 5.x version since it would take enormous amount of testing which is not available for the stable products.

Given all the above and the fact that the issue normally occurs very rarely (no occurrences for the last year, ~100 SWARM runs with ~150 full cluster deployments in each SWARM run) and is mostly connected with overloaded H/W or VM resources I'm marking it as Won't Fix. Please add more resources to your deployments and troubleshoot high load on your environments.