Disk resource consumption is inconsistent between scheduler and resource tracker

Bug #1550098 reported by Yingxin on 2016-02-26
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Medium
Yingxin

Bug Description

The way scheduler consumes disk resources in host state is inconsistent with RT's way in compute service.

The scheduler consumes "free_disk_mb" in its host state.
https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L266

It comes from the min value of "free_disk_gb" and "disk_available_least" in ComputeNode object.
https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L189-L201

But compute node changes "local_gb_used" instead in consuming resources from a request. https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L709-L710

There is a inconsistent gap(confirmed) of compute node state between scheduler and resource tracker, but somehow the compute node will be updated by virt driver to the consistent status after maybe 10 seconds.

Yingxin (cyx1231st) on 2016-02-26
Changed in nova:
assignee: nobody → Yingxin (cyx1231st)
status: New → In Progress
Changed in nova:
status: In Progress → Confirmed
Yingxin (cyx1231st) wrote :

Change to Medium because it is a prerequisite of future schedulers such as resource-provider scheduler and shared-state scheduler.

Changed in nova:
importance: Undecided → Medium
Changed in nova:
status: Confirmed → In Progress
Yingxin (cyx1231st) on 2016-07-06
tags: added: scheduler

Hello @Yingxin:

I've seen this error in gate-tempest-dsvm-neutron-identity-v3-only-full-ubuntu-xenial-nv too.

Do you have a fix for this problem? I don't want to step over your solution on this problem. If you are stuck on this bug, do you want to transfer it to me?

A possible solution: to test in https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L205 if free_gb is None. If it is, assign free_gb = least_gb. What do you think?

Yingxin (cyx1231st) wrote :

Hi Rodolfo,

I had a fix in https://review.openstack.org/#/c/290302/ long ago. I think it still works. It moves the "least_disk_mb" calculation to disk_filter, so free_disk_mb in host manager is now consistent with the resource tracker "free_disk_gb".

If it doesn't fix the error in gate-tempest-dsvm-neutron-identity-v3-only-full-ubuntu-xenial-nv, feel free to upload your version :)

Sean Dague (sdague) wrote :

There are no currently open reviews on this bug, changing
the status back to the previous state and unassigning. If
there are active reviews related to this bug, please include
links in comments.

Changed in nova:
status: In Progress → Confirmed
assignee: Yingxin (cyx1231st) → nobody
Sean Dague (sdague) on 2017-06-28
Changed in nova:
assignee: nobody → Yingxin (cyx1231st)
status: Confirmed → In Progress
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers