Post-migration, Cinder volumes lose disk cache value, resulting in I/O latency
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Kashyap Chamarthy | ||
Newton |
Fix Committed
|
Medium
|
Kashyap Chamarthy | ||
Ocata |
Fix Committed
|
Medium
|
Tony Breeds |
Bug Description
Description
===========
[This was initially reported by a Red Hat OSP customer.]
The I/O latency of a Cinder volume after live migration of an instance
to which it's attached increases significantly. This stays increased
till the VM is stopped and started again. [VM is booted with Cinder
volume.
This is not the case when using a disk from a Nova store backend [
without Cinder volume] -- or at least the difference isn't so
significantly high after a live migration.
The storage backend is Ceph 2.0.
How reproducible: Consistently
Steps to Reproduce
==================
(0) Both the Nova instances and Cinder volumes are located on Ceph
(1) Create a Nova instance with a Cinder volume attached to it
(2) Live migrate it to a target Compute node
(3) Run `ioping` (`ioping -c 10 .`) on the Cinder volume.
Alternatively, run other I/O benchmarks like using `fio` with
'direct=1' (which uses non-bufferred I/O) as a good sanity check to
get a second opinion regarding latency.
Actual result
=============
Before live migration: `ioping` output on the Cinder volume attached to a Nova
instance:
[guest]$ sudo ioping -c 10 .
4 KiB <<< . (xfs /dev/sda1): request=1 time=98.0 us (warmup)
4 KiB <<< . (xfs /dev/sda1): request=2 time=135.6 us
4 KiB <<< . (xfs /dev/sda1): request=3 time=155.5 us
4 KiB <<< . (xfs /dev/sda1): request=4 time=161.7 us
4 KiB <<< . (xfs /dev/sda1): request=5 time=148.4 us
4 KiB <<< . (xfs /dev/sda1): request=6 time=354.3 us
4 KiB <<< . (xfs /dev/sda1): request=7 time=138.0 us (fast)
4 KiB <<< . (xfs /dev/sda1): request=8 time=150.7 us
4 KiB <<< . (xfs /dev/sda1): request=9 time=149.6 us
4 KiB <<< . (xfs /dev/sda1): request=10 time=138.6 us (fast)
--- . (xfs /dev/sda1) ioping statistics ---
9 requests completed in 1.53 ms, 36 KiB read, 5.87 k iops, 22.9 MiB/s
generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s
min/
After live migration, `ioping` output on the Cinder
[guest]$ sudo ioping -c 10 .
4 KiB <<< . (xfs /dev/sda1): request=1 time=1.03 ms (warmup)
4 KiB <<< . (xfs /dev/sda1): request=2 time=948.6 us
4 KiB <<< . (xfs /dev/sda1): request=3 time=955.7 us
4 KiB <<< . (xfs /dev/sda1): request=4 time=920.5 us
4 KiB <<< . (xfs /dev/sda1): request=5 time=1.03 ms
4 KiB <<< . (xfs /dev/sda1): request=6 time=838.2 us
4 KiB <<< . (xfs /dev/sda1): request=7 time=1.13 ms (slow)
4 KiB <<< . (xfs /dev/sda1): request=8 time=868.6 us
4 KiB <<< . (xfs /dev/sda1): request=9 time=985.2 us
4 KiB <<< . (xfs /dev/sda1): request=10 time=936.6 us
--- . (xfs /dev/sda1) ioping statistics ---
9 requests completed in 8.61 ms, 36 KiB read, 1.04 k iops, 4.08 MiB/s
generated 10 requests in 9.00 s, 40 KiB, 1 iops, 4.44 KiB/s
min/
This goes back to an average of 200us again after shutting down and
starting up the instance.
Expected result
===============
No I/O latency experienced on Cinder volumes.
Changed in nova: | |
assignee: | nobody → Kashyap Chamarthy (kashyapc) |
status: | New → Confirmed |
Changed in nova: | |
status: | Confirmed → In Progress |
Changed in nova: | |
importance: | Undecided → Medium |
Here's the change to fix this issue:
https:/ /review. openstack. org/#/c/ 485752/ -- libvirt: Post-migration,
set cache value for Cinder volume(s)