scheduler hints should persist with instance for use in migration, resize, and evacuate

[Tom Gall]

If you've used a scheduler hint such as picking resources that are in close proximity to a particular ip address.

ex:
scheduler_hints': { 'cidr': '/32',
                                'build_near_host_ip': affinity_ip}

Now you want to migrate to a different host. At no point during the migration will the hints be revisited. The original hints are not even saved!

On migration the hints should be applied against the destination. If the destination does no pass the filter an (overridable?) error should be raised.

I have an initial implementation that can be found at : git://gitorious.org/nova/nova.git
                                                                               branch: respect_scheduler_hints_during_migration

I'll submit the patches via review.openstack.org.

[Mark McLoughlin (markmc)]

Patch is here:

  http://gitorious.org/nova/nova/commit/cbafdcb559acc05d93283163ae05c26b7123a57f

Interesting one - since the admin has to choose a destination host, I guess the admin is manually scheduling the instance. Do we want to prevent admins from choosing a host that doesn't match the initial scheduler hints or any other filter criteria?

Marking as confirmed and assigning to you. Looking forward to you submitting the patch. Thanks!

[Tom Gall (tom-gall)]

Yes, exactly. On migration if one tries to migrate to a host which wouldn't have passed the filter than raise an error. It does beg the question that it should/could be overridden.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/12148

[Thierry Carrez (ttx)]

Looks like you're not working on that anymore. Please set back to InProgress and reassign to you if you're working on proposing a change for merging.

[Marco CONSONNI (marco-consonni)]

Related with this issue.

I configured the scheduler by adding RamFilter to scheduler_default_filters in nova.conf.
Then I set ram_allocation_ratio to 1.5.

These settings force the scheduler to calculate the available RAM on a host as follows:

 available-RAM = physical-RAM * 1.5

In other words, the scheduler behaves like the RAM on a host is 1.5 times the actual RAM.

When I boot instances, I see that they are scheduled to a host till it runs out of available-RAM (more than physical-RAM).
However, when I migrate an instance to a host, the check is made against the physical-RAM *NOT* against the available-RAM.
In other words the behavior of the scheduler is not the same when I boot and when I migrate.

I looked into the code and it seems to me that this is really a design flaw.
As a mater of fact, when migrating only the RAM limit is taken account albeit the calculation is wrong, as reported above.
Other limits are simply ignored (e.g. CoreFilter).

[Marco CONSONNI (marco-consonni)]

More thoughts on this aspect.

Should scheduler's filters to be taken into account when migrating?
To me, this is the real question.

In my previous post I listed a couple of examples (RamFilter and CoreFilter) but what about other filters?
Do we have to take them in account when we migrate or not?

And what happens if we try to migrate to a "disabled" compute host (nova-manage service disable --service=nova-compute --host=<host>)?

At the moment you can do it provided that the host is up and running and that kvm is working.

But is this a sound behavior?

[Russell Bryant (russellb)]

A related change to this ...

https://review.openstack.org/#/c/22193

It doesn't fix this particular scenario, where people are saying that the scheduler's logic should be taken into account even when a specific target host is specified.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/45450

[Tracy Jones (tjones-i)]

this has not been touched in a long time and the patches are abandoned. Please set back to in progress if you start to work on it

[Ryan Hooper (ryhooper)]

I have ran into the same issue with the live-migratation command when I launched a guest with a trusted filter. My feelings are that the admin shouldn't be able to move a vm to a host that doesn't meet the filter requirements. My main question is should the admin have the right to bypass the filters? Is that the reason why this was never committed?

[Marco CONSONNI (marco-consonni)]

Same behaviour when you evacuate a host.

Considering that scheduler hints are gaining a lot of attention (see NUMA policies for supporting NFV workloads https://wiki.openstack.org/wiki/VirtDriverGuestCPUMemoryPlacement), I think that this bug needs to be taken into account

[Boden R (boden)]

+1
Would love to see some focus on this and moreover see the functionality carried over into cinder w/r/t migration of vols.

[Boden R (boden)]

BTW -- I could also see the use in supporting the ability to pass in new hints on the actual migrate command / api.. e.g. do migration and use these new hints when you do placement.

[Chris Friesen (cbf123)]

Pretty sure that NUMA topology, vCPU pinning, and hugepage support all depend on this stuff, no? So if you boot an instance that requires any of that functionality and we don't apply them when migrating/resizing/evacuating then that makes those features pretty much useless.

[Markus Zoeller (markus_z) (mzoeller)]

This wishlist bug has been open a year without any activity. I'm going to move it to "Opinion / Wishlist", which is an easily-obtainable queue of older requests that have come on. This bug can be reopened (set back to "New") if someone decides to work on this.