Treating SSD like HDD (i/o scheduler).

[Piotr Skamruk]

In deployed by fuel nodes there is no distinction in HDD vs SSD configuration.
Both have in /sys/block/sd?/queue/scheduler default 'cfq' scheduler, which on SSD drives gives ~20% less performance (IOPS) than 'noop', or 'deadline'.

Question is how to distinct and set this?

Under newer linux kernels than used as default 2.6.32 (starting probably from 2.6.39) it's easy:
(from https://wiki.archlinux.org/index.php/Solid_State_Drives#Using_udev_for_one_device_or_HDD.2FSSD_mixed_environment)
/etc/udev/rules.d/60-schedulers.rules
# set deadline scheduler for non-rotating disks
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"

On older kernels (as used in Centos 6.5, used in Fuel 6.o) which do not have /sys/block/sd?/queue/rational - this could be done by:
for dev in /dev/sd? ; do smartctl -a $dev | grep -q SSD || echo deadline >/sys/block/${dev##*/}/queue/scheduler ; done

... if your drive has "SSD" somewhere in smartctl (not installed by default) output (i don't know how other manufacturers have, but it's true at least for Intel drives).

[Alexander Nevenchannyy (anevenchannyy)]

This is bug affect only CentOS deployments.
Ubuntu Server use deadline by default.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/161685

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/162039

[Piotr Skamruk (jell)]

Copy of comment from gerrit:

I think that this is a wrong way to solve this issue. In this way You are changing scheduler in every situation (also for HDD drives) what is not exactly what I reported.
But if You have tests which confirms that there is no degradation in HDD (or mixed) setups - go ahead and use this patch. I have opinion from my colleagues that this probably will provide degradation on used HDD.
I did not tested this, so I can't add +1/-1 on this.
If You have test results, again - go ahead. If not - at least try fio on HDD with CFQ vs deadline, and write about this in there as comment.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/162107

[OpenStack Infra (hudson-openstack)]

Change abandoned by Bart?omiej Piotrowski (<email address hidden>) on branch: stable/6.0
Review: https://review.openstack.org/162039
Reason: Wrong Change-Id; https://review.openstack.org/162107

[Alexander Nevenchannyy (anevenchannyy)]

jell, as i'm wroted above.

CentOS use CFQ by default.
Ubuntu server - use deadline by default at lest three years.

So at this moment we have, that all ubuntu deployments use deadline and we don't have any problems with it's
About performance of deadline vs cfq. You may to read this for example:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html-single/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/

Load profile databases close enough to the load profile virtualization (many random access with random block size in many threads)

[Piotr Skamruk (jell)]

Sounds resonable to me.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/161685
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=cf3d193fad5eee3ff345379ced42ad7be8e4b6b2
Submitter: Jenkins
Branch: master

commit cf3d193fad5eee3ff345379ced42ad7be8e4b6b2
Author: Alexander Nevenchannyy <email address hidden>
Date: Thu Mar 5 14:21:27 2015 +0300

    Add elevator parameter to startup kernel options, and set it to deadline.

    This is change linux i/o scheduler to deadline, instead of cfq.

    Change-Id: Iff947f0053577f19441c04101e5a35a7820e40a0
    Closes-bug: #1418126

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/162107
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=ec917ae36d2a1b482b189f29483cfdbc5ce700be
Submitter: Jenkins
Branch: stable/6.0

commit ec917ae36d2a1b482b189f29483cfdbc5ce700be
Author: Bartłomiej Piotrowski <email address hidden>
Date: Fri Mar 6 08:11:11 2015 +0100

    Change default IO scheduler to deadline

    The default CFQ scheduler is mostly geared towards desktop usage.

    Change-Id: Iff947f0053577f19441c04101e5a35a7820e40a0
    Closes-bug: #1418126

[Alexander Nevenchannyy (anevenchannyy)]

After deployment lab from ISO with this patch https://review.openstack.org/#/c/161685/ i'm found that it's not resolve our problem. So new status in progress

[Piotr Skamruk (jell)]

I can also confirm that in our internal tests with this change made manually - we observed speed degradation.
For this moment I can not give there any results, but if it's necessary I can share some results in monday.

[Alexander Nevenchannyy (anevenchannyy)]

Jell, please attach your tests.

[Piotr Skamruk (jell)]

dd speed on vm volume (ceph as backend):
        block size old settings deadline in kernel parameters
read 4k 3.7 MB/s 2.9 MB/s
        8k 7.2 MB/s 5.9 MB/s
        64k 46.9 MB/s 39.5 MB/s
        512k 172 MB/s 173 MB/s
        1024k 260 MB/s 240 MB/s
write 4k 50.6 MB/s 50.0 MB/s
        8k 97.0 MB/s 95.2 MB/s
        64k 244 MB/s 231 MB/s
        512k 272 MB/s 249 MB/s
        1024k 272 MB/s 242 MB/s

[Piotr Skamruk (jell)]

As formatting is lost on launchpad - results in attachment.

[Bartłomiej Piotrowski (bpiotrowski)]

Revert for 6.1: https://review.openstack.org/#/c/175297/
Revert for 6.0: https://review.openstack.org/#/c/175296/

[Dmitry Mescheryakov (dmitrymex)]

I've discussed the bug with Alexander Nevenchannyy. He says that we can't just change set scheduler to some 'hardcoded' value: it works differently for different environments. I am setting the bug as won't fix for 6.1. Later we will convert it to blueprint and consider for future releases.

[Bogdan Dobrelya (bogdando)]

Please explain the current deployment and operations impact of this bug - is it really high? Why it was raised from medium?