Magnum conductor performs periodic tasks in order to synchronise magnum state with that of other services, such as heat. This occurs with interval as set by [DEFAULT] periodic_interval_max, default 60.
The magnum conductor service forks a number of worker processes to handle RPC requests. The number of workers is determined by [conductor] workers, or the number of CPU cores if unset.
Based on the magnum conductor logs, each of the workers runs periodic tasks, and they all seem to be temporally aligned. For example, if a cluster fails during creation due to the famous 'no valid hosts found' failure, there will generally be many of these messages in the logs:
2017-07-04 19:41:04.454 53 ERROR magnum.drivers.heat.driver [req-a9a2942d-ef71-41b3-83b0-f8a4ac5f90f5 - - - - -] Cluster error, stack status: CREATE_FAILED, stack_id: 6549d2e2-d4a8-47ad-b07e-bb28a0b5a0a1, reason: Resource CREATE failed: ResourceInError: resources.kube_masters.resources[0].resources.kube-master: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
Clearly this is not ideal, as there are many workers performing the same task at the same time, all with the same outcome, which includes spamming the logs with multiple identical error messages.
The periodic task execution framework should be updated to distribute tasks among conductor services on multiple hosts, and multiple worker processes within each conductor service.
This does not occur to the same degree on ocata, as multi-worker support[1] was added during the pike development cycle. In ocata, conductor services running on multiple hosts will still all run the same periodic update jobs, which seems inefficient, but not nearly as bad as having multiple workers running periodic updates.
A simple solution that would at least revert to the ocata behaviour would be to only execute periodic updates in the parent process, and not the workers.
[1] https:/