scheduler selects the same ironic node several times

[Pavlo Shchelokovskyy]

Observed on ironic multinode grenade job (using Ocata scheduler).

Ironic returns its nodes in the same relative order (by internal DB id). Quite often (and in DevStack always by default) ironic nodes are identical, thus filter scheduler gives them the same weight. As a result, during concurrent requests to schedule instances, the weighed_hosts list goes always in the same order and is being always consumed from the start.

This leads to the first node selected often enough to exceed the default number of retries when it is being stolen by another concurrent request (as it also always picks the first one from the list).

See log examples from the same gate job [0-2], failure [3] (ServerActionsTestJSON test failure). Notice how the weighed hosts list is always is always in the same order, and scheduler retries 3 times on nodes that are being already occupied by another parallel request, always picking the currently first one.

This could be fixed by increasing the host_subset_size config option from its default value of 1,
which would bring some randomness to the first element.
While fine (and actually recommended) for baremetal-only case, this choice is a bit suboptimal in a mixed hypervizor (virtual+ironic computes) as it makes scheduling logic for virtual computes less ideal.

Instead, it might be better to always randomize the first hosts in the weighed_hosts list for hosts with identical (and maximal) weight as those should be equally good candidates to schedule to. This will decrease collision and rescheduling chances, definitely for ironic nodes, but also to some tiny extent for standard compute hosts as well.

[0] http://logs.openstack.org/12/493812/2/check/gate-grenade-dsvm-ironic-multinode-multitenant-ubuntu-xenial/8d3f840/logs/old/screen-n-sch.txt.gz#_2017-08-15_13_27_29_410

[1] http://logs.openstack.org/12/493812/2/check/gate-grenade-dsvm-ironic-multinode-multitenant-ubuntu-xenial/8d3f840/logs/old/screen-n-sch.txt.gz#_2017-08-15_13_27_31_839

[2] http://logs.openstack.org/12/493812/2/check/gate-grenade-dsvm-ironic-multinode-multitenant-ubuntu-xenial/8d3f840/logs/old/screen-n-sch.txt.gz#_2017-08-15_13_27_34_244

[3] http://logs.openstack.org/12/493812/2/check/gate-grenade-dsvm-ironic-multinode-multitenant-ubuntu-xenial/8d3f840/logs/testr_results.html.gz

[Pavlo Shchelokovskyy (pshchelo)]

patch proposed https://review.openstack.org/#/c/494136/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/494136
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3759f105a7c4c3029a81a5431434190ef1bbb020
Submitter: Zuul
Branch: master

commit 3759f105a7c4c3029a81a5431434190ef1bbb020
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Wed Aug 16 08:50:49 2017 +0000

    Allow shuffling hosts with the same best weight

    This patch adds a new boolean config option
    `[filter_scheduler]shuffle_best_same_weighed_hosts` (default False).

    Enabling it will improve scheduling for the case when host_subset_size=1
    but list of weighed hosts contains many hosts with the same best weight
    (quite often the case for ironic nodes).
    On the other hand, enabling it will also make VM packing on hypervisors
    less dense even when host weighing is completely disabled.

    Change-Id: Icee137e15f264da59a1bdc1dc1ecfeaac82b98c6
    Closes-Bug: #1711184

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/nova 17.0.0.0b2 development milestone.