Race condition between RT and scheduler

[Radoslav Gerganov]

The HostState object which is used by the scheduler is using the 'stats' property of the compute node to derive its own values, e.g. :

    self.stats = compute.stats or {}
    self.num_instances = int(self.stats.get('num_instances', 0))
    self.num_io_ops = int(self.stats.get('io_workload', 0))
    self.failed_builds = int(self.stats.get('failed_builds', 0))

These values are used for both filtering and weighing compute hosts. However, the 'stats' property of the compute node is cleared during the periodic update_available_resources() and populated again. The clearing occurs in RT._copy_resources() and it preserves only the old value of 'failed_builds'. This creates a race condition between RT and scheduler which may result into populating wrong values for 'num_io_ops' and 'num_instances' into the HostState object and thus leading to incorrect scheduling decisions.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/611852

[Radoslav Gerganov (rgerganov)]

I just found that this problem is fixed in the master branch as part of bug #1729621. However, it is not backported to stable releases.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Radoslav Gerganov (<email address hidden>) on branch: master
Review: https://review.openstack.org/611852
Reason: https://review.openstack.org/#/c/520024/ is a better fix for this