Ubuntu server uses CFQ scheduler instead of deadline

[Hark]

Ubuntu server now uses the '-generic' kernel, as the '-server' kernel doesn't exist anymore. With this change the default server disk scheduler changed from deadline to CFQ. This is very bad for servers, deadline should absolutely be the default for servers.

For example we moved some KVM images to an Ubuntu 12.04 server, but we had to revert as we got lots of disk errors and eventually frozen disks on these KVM's due to write timeouts. It took us a while before we found out that this 12.04 server used CFQ, which surprised us a lot.

[Brad Figg (brad-figg)]

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1008400

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Hark (ubuntu-komkommerkom)]

Log files are completely irrelevant for this issue.

[Leann Ogasawara (leannogasawara)]

I believe you can boot with "elevator=deadline" to switch back to the Deadline I/O Scheduler.

[Bryan Wu (cooloney)]

I tried install latest Ubuntu Server image on my machine and confirmed that we are using CFQ for default IO scheduler for both server and desktop now.

Hark, we merged -server and -generic kernel image together for easier maintenance and most difference can be switched via kernel command line or user space interface. As Leann said, we can add "elevator=deadline" in kernel boot command line to use Deadline I/O scheduler by default. Or just change /sys/block/DEV/queue/scheduler from CFQ to deadline.

If it's necessary, I'm think about add some script to do this in our server image. But that probably related to grub, upstart init script or other user space utils not kernel issue now.

-Bryan

[Hark (ubuntu-komkommerkom)]

Yes Leann and Bryan, I can boot with "elevator=deadline" and that is exactly what I did after I noticed that CFQ was being used. But this remains a bug that should be solved. Most people don't even know what a scheduler is, or how to change it. On a server it is almost always a bad idea to use CFQ. Brian, your suggestion to add a script in the server image sounds good to me, do you know how to get this to the right people?

[Andy Whitcroft (apw)]

@Hark -- though the default is changed we would not expect hangs from using CFQ, or are we just talking about gross unfairness. Can you share the logs of the failures.

[Hark (ubuntu-komkommerkom)]

@Andy - The logs that could be interesting are the logs of the virtual server (kvm). However I don't have this log anymore, as I had to replace the kvm image with a backup because of this problem. But let me try to explain the situation, you'll see that unfairness is not the issue here.

- I installed a new Ubuntu 12.04 server (host), with CFQ as scheduler (new default)
- A kvm image was placed on the host (local disks, software raid), and it was started. I call this virtual server 'kvm'.
- The kvm was running fine all day, no problems at all
- At night some big files were rsync'd to the host, with normal priorities.
- A few minutes after the rsync started the kvm started logging ata disk timeouts. After some time the disks were remounted read-only due to these errors.
- The next day I tried to restart the kvm, but the image was completely corrupted.
- After placing back a backup the kvm was started again
- The next night exactly the same happened during the rsync.
- The next day I placed back the backup again, and changed the scheduler to 'deadline' (old default). This solved the problem.

But this is just one of many examples of what can go wrong with CFQ. Internet is full of stories about CFQ causing very slow i/o or worse for several common server tasks. I really think the default scheduler should be changed to deadline again for servers. It is not without reason that the old -server kernels did this.

[Stefan Bader (smb)]

As Andy said the wrong scheduler decision should worst case result in worse performance. But getting so bad that a kvm guest sees disk failures is more than should be allowed. It would be good if we could reproduce this. For that it would be good to know more about the setup:

For the host:
- What kind of RAID (I assume md but which level) is used?
- Which file system and is that on the RAID directly or on LVM-LVs?

For the guest:
- Is the virtual disk mapped to a image file or a partition/LV?
- is it defined as a virtio disk or an emulated disk?

Then the rsync: is that from inside the guest to the host? Or from another server to the host? Depending on the setup is the target on the same fs or not?

[Hark (ubuntu-komkommerkom)]

For the host:
-We use a md RAID indeed, raid 1
-The host and the guest both use ext4. On the host the file system is on a LVM LV indeed

For the guest:
-The virtual disk is in a qcow2 image file
-At first we used an emulated IDE disk. We have now changed this to virtio, but this was done a few days after the problems had been solved by changing the scheduler to deadline. Maybe using IDE emulation helps to reproduce the problem, I don't know.

The rsync was done from another server to the host, not with increased priority or anything like that.

I have to admit that our setup was not optimal. The disks are modern but quite slow, and the RAID 1 config and IDE emulation doesn't help either. So I am not surprised this could be quite slow, however disk starvation inside the guest should not happen. As said, changing only the scheduler solved this problem completely.

[Stefan Bader (smb)]

Thanks for the detailed info. I agree that disk starvation should not happen. Though it could be that deadline just about covers some other problem. There are a few pieces that add to slow down the host io, the RAID1 and the qcow2 file image. Using the IDE emulation would actually slow down the guest. But, and I am speculating here a bit, having the rsync going on while the guest writes to the virtual disk may just get delayed a bit over the timeout that the IDE driver expects some answer from the emulated drive. And if that is the case then there could be some potential of this to happen regardless of the scheduler choice. It just happens to help in your case.
I did not see the same problems on a local setup with cfq, but then I had a raw image and were using virtio. I will try to adapt this setup to at least having qcow2 and emulated disks to see whether that reproduces the problems. If possible I would like to keep my RAID5 the way it is. ;)

[Hark (ubuntu-komkommerkom)]

Thinking about it the IDE emulation may be the problem. If I remember it well the disk timeout messages on the guest had 'ata' in it. This is not really my cup of tea but I know that ATA and IDE are related.

But please remember that my problem is only one example of what can happen when using CFQ on servers. For lots of common server tasks using CFQ is not recommended, so I still think this is a valid bug report.

[Bryan Wu (cooloney)]

Hark,

We (Ubuntu kernel team) have been evaluating CFQ vs deadline. For this specific bug report, we need some hard facts to make a decision about changing to deadline IO scheduler. For example, any test case or sequence we can reproduce easily.

Do you have such simple test case sharing with us? We are also trying to collect some data right now.

-Bryan

[Hark (ubuntu-komkommerkom)]

Hi Bryan,

Thanks for taking this seriously. I only have the testcase I described above. This server is now in production, so I cannot do any more tests on it.
However it should be easy to test the performance of deadline vs CFQ on a server, and lots of people did this and wrote about it on their blogs/websites. But this is only the performance of course, for more serious problems you could resemble my scenario;start a KVM with ide emulation on a host with a normal (slow) disk, do regular writes on the KVM and do a write on the host that lasts at least several minutes on the same physical disk. I used a Dell R300 server with two WD Caviar Green disks, you should be able to get the same result on most other hardware configurations, as long as you don't use too fast disks, hardware raid controllers etc.

[Bryan Wu (cooloney)]

Hark,

We've changed default IO scheduler from CFQ to deadline for both server and desktop kernel. Please find our latest kernel from git tree for Quantal release, any test and bug report are welcome.

Thanks,
-Bryan

[Sergey Alekhin (salekhin)]

>>We (Ubuntu kernel team) have been evaluating CFQ vs deadline. For this specific bug report, we need some hard facts to make a >>decision about changing to deadline IO scheduler. For example, any test case or sequence we can reproduce easily.

Here is comparison of CFQ vs Deadline
http://publib.boulder.ibm.com/infocenter/lnxinfo/v3r0m0/topic/liaat/liaatbpdeadline.htm

[Jakob Unterwurzacher (jakobunt)]

Well the obvious downside of deadline is that priorities (ionice) are not supported.

[Hark (ubuntu-komkommerkom)]

Thanks for fixing this in Quantal! Deadline is the 'best' choice for servers and computers with SSD's. However there are situations where CFQ is preferred, like desktop PC's with a hard drive. But I can imagine it isn't easy to detect this in a reliable way.

[Alex Bligh (ubuntu-alex-org)]

See also bug:
 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1064521

[Ivan Baldo (ibaldo)]

In my humble opinion, several kernel options should be tested before each new Ubuntu version to see what is the best one at the time.
This is one such option.
Isn't the CFQ scheduler in better shape in Ubuntu 14.04?
Maybe not, I am just asking :-).
Thanks for the info! :-)

[Ivan Baldo (ibaldo)]

BTW, that IBM article... it doesn't mention what versions they used, which I think is very important :-(.

[Victor Zverovich (victor-zverovich-net)]

I observe substantial lag under heavy I/O (in my case I was unzipping a multigigabyte archive) with the deadline scheduler. Manually switching to the cfq scheduler improved responsiveness and eliminated the lag. Perhaps cfq is a better choice for the desktop version?

[Bjoern (bjoern-t)]

In my opinion I also agree to use deadline scheduler for server kernel which is also the same problem on the 4.15 kernel, if a server kernel would still exist.

Under trusty we generally saw good performance with deadline scheduler and on a recent SSD based deployment with 4.15 I noticed a large performance regression for random IO until we reverted to deadline. You could argue that the scheduler can be changed at any time but on the contrary I would assume more general suitable parameters from a server distro vs a desktop distro.
My larger problem is that this change was seemingly done via https://kernel.ubuntu.com/git/ubuntu/ubuntu-zesty.git/commit/?id=af80b83a2b6184fea27f050948146fcd9a28070d by a test based on kernel compiles etc, which to be honest, is not a standard server use case. Apart from the issue that I could not spot any documentation on the release notes that this has changed. If this has been documented and simply missed it, then please ignore this comment. But now I'm wondering what else has changed now.