Treating SSD like HDD (i/o scheduler).

Bug #1418126 reported by Piotr Skamruk on 2015-02-04
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Medium
Fuel Library (Deprecated)
5.1.x
Medium
MOS Maintenance
6.0.x
Medium
Bartłomiej Piotrowski
6.1.x
Medium
Alexander Nevenchannyy

Bug Description

In deployed by fuel nodes there is no distinction in HDD vs SSD configuration.
Both have in /sys/block/sd?/queue/scheduler default 'cfq' scheduler, which on SSD drives gives ~20% less performance (IOPS) than 'noop', or 'deadline'.

Question is how to distinct and set this?

Under newer linux kernels than used as default 2.6.32 (starting probably from 2.6.39) it's easy:
(from https://wiki.archlinux.org/index.php/Solid_State_Drives#Using_udev_for_one_device_or_HDD.2FSSD_mixed_environment)
/etc/udev/rules.d/60-schedulers.rules
# set deadline scheduler for non-rotating disks
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"

On older kernels (as used in Centos 6.5, used in Fuel 6.o) which do not have /sys/block/sd?/queue/rational - this could be done by:
for dev in /dev/sd? ; do smartctl -a $dev | grep -q SSD || echo deadline >/sys/block/${dev##*/}/queue/scheduler ; done

... if your drive has "SSD" somewhere in smartctl (not installed by default) output (i don't know how other manufacturers have, but it's true at least for Intel drives).

Changed in fuel:
assignee: nobody → Bartlomiej Piotrowski (bpiotrowski)
status: New → Confirmed
Changed in fuel:
importance: Undecided → Medium
milestone: none → 6.1

This is bug affect only CentOS deployments.
Ubuntu Server use deadline by default.

Fix proposed to branch: master
Review: https://review.openstack.org/161685

Changed in fuel:
assignee: Bartlomiej Piotrowski (bpiotrowski) → Alexander Nevenchannyy (anevenchannyy)
status: Confirmed → In Progress
Piotr Skamruk (jell) wrote :

Copy of comment from gerrit:

I think that this is a wrong way to solve this issue. In this way You are changing scheduler in every situation (also for HDD drives) what is not exactly what I reported.
But if You have tests which confirms that there is no degradation in HDD (or mixed) setups - go ahead and use this patch. I have opinion from my colleagues that this probably will provide degradation on used HDD.
I did not tested this, so I can't add +1/-1 on this.
If You have test results, again - go ahead. If not - at least try fio on HDD with CFQ vs deadline, and write about this in there as comment.

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/162107

Change abandoned by Bart?omiej Piotrowski (<email address hidden>) on branch: stable/6.0
Review: https://review.openstack.org/162039
Reason: Wrong Change-Id; https://review.openstack.org/162107

jell, as i'm wroted above.

CentOS use CFQ by default.
Ubuntu server - use deadline by default at lest three years.

So at this moment we have, that all ubuntu deployments use deadline and we don't have any problems with it's
About performance of deadline vs cfq. You may to read this for example:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html-single/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/

Load profile databases close enough to the load profile virtualization (many random access with random block size in many threads)

Piotr Skamruk (jell) wrote :

Sounds resonable to me.

Reviewed: https://review.openstack.org/161685
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=cf3d193fad5eee3ff345379ced42ad7be8e4b6b2
Submitter: Jenkins
Branch: master

commit cf3d193fad5eee3ff345379ced42ad7be8e4b6b2
Author: Alexander Nevenchannyy <email address hidden>
Date: Thu Mar 5 14:21:27 2015 +0300

    Add elevator parameter to startup kernel options, and set it to deadline.

    This is change linux i/o scheduler to deadline, instead of cfq.

    Change-Id: Iff947f0053577f19441c04101e5a35a7820e40a0
    Closes-bug: #1418126

Changed in fuel:
status: In Progress → Fix Committed

Reviewed: https://review.openstack.org/162107
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ec917ae36d2a1b482b189f29483cfdbc5ce700be
Submitter: Jenkins
Branch: stable/6.0

commit ec917ae36d2a1b482b189f29483cfdbc5ce700be
Author: Bartłomiej Piotrowski <email address hidden>
Date: Fri Mar 6 08:11:11 2015 +0100

    Change default IO scheduler to deadline

    The default CFQ scheduler is mostly geared towards desktop usage.

    Change-Id: Iff947f0053577f19441c04101e5a35a7820e40a0
    Closes-bug: #1418126

Changed in fuel:
status: Fix Committed → Fix Released
Changed in fuel:
status: Fix Released → Fix Committed

After deployment lab from ISO with this patch https://review.openstack.org/#/c/161685/ i'm found that it's not resolve our problem. So new status in progress

Changed in fuel:
status: Fix Committed → In Progress
Piotr Skamruk (jell) wrote :

I can also confirm that in our internal tests with this change made manually - we observed speed degradation.
For this moment I can not give there any results, but if it's necessary I can share some results in monday.

Jell, please attach your tests.

Piotr Skamruk (jell) wrote :

dd speed on vm volume (ceph as backend):
        block size old settings deadline in kernel parameters
read 4k 3.7 MB/s 2.9 MB/s
        8k 7.2 MB/s 5.9 MB/s
        64k 46.9 MB/s 39.5 MB/s
        512k 172 MB/s 173 MB/s
        1024k 260 MB/s 240 MB/s
write 4k 50.6 MB/s 50.0 MB/s
        8k 97.0 MB/s 95.2 MB/s
        64k 244 MB/s 231 MB/s
        512k 272 MB/s 249 MB/s
        1024k 272 MB/s 242 MB/s

Piotr Skamruk (jell) wrote :

As formatting is lost on launchpad - results in attachment.

Changed in fuel:
status: In Progress → Triaged
milestone: 6.1 → 7.0
Changed in fuel:
milestone: 7.0 → 6.1
status: Triaged → New
Changed in fuel:
status: New → Won't Fix
Changed in fuel:
milestone: 6.1 → 7.0
status: Won't Fix → Triaged
no longer affects: fuel/7.0.x
Changed in fuel:
milestone: 7.0 → 6.1
status: Triaged → New
Changed in fuel:
status: New → Confirmed
Changed in fuel:
importance: Medium → High

I've discussed the bug with Alexander Nevenchannyy. He says that we can't just change set scheduler to some 'hardcoded' value: it works differently for different environments. I am setting the bug as won't fix for 6.1. Later we will convert it to blueprint and consider for future releases.

Bogdan Dobrelya (bogdando) wrote :

Please explain the current deployment and operations impact of this bug - is it really high? Why it was raised from medium?

Changed in fuel:
status: Confirmed → Won't Fix
no longer affects: fuel/7.0.x
Changed in fuel:
milestone: 6.1 → 7.0
status: Won't Fix → Triaged
importance: High → Medium
Igor Shishkin (teran) on 2015-10-08
Changed in fuel:
milestone: 7.0 → 8.0
Changed in fuel:
assignee: Alexander Nevenchannyy (anevenchannyy) → nobody
assignee: nobody → Fuel for Openstack (fuel)
Changed in fuel:
assignee: Fuel for Openstack (fuel) → Fuel Library Team (fuel-library)
tags: added: feature
Changed in fuel:
status: Triaged → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Bug attachments