| 2014-03-25 21:37:27 |
Holger Mauermann |
bug |
|
|
added bug |
| 2014-03-25 21:37:27 |
Holger Mauermann |
attachment added |
|
disk latency graph after upgrading to trusty https://bugs.launchpad.net/bugs/1297522/+attachment/4043602/+files/vda-month.png |
|
| 2014-03-25 22:00:59 |
Brad Figg |
linux (Ubuntu): status |
New |
Incomplete |
|
| 2014-03-25 22:01:00 |
Brad Figg |
tags |
|
trusty |
|
| 2014-03-25 22:32:14 |
Holger Mauermann |
attachment added |
|
apport.linux.WwkgUu.apport https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1297522/+attachment/4043636/+files/apport.linux.WwkgUu.apport |
|
| 2014-03-25 22:32:35 |
Holger Mauermann |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2014-03-26 18:33:46 |
Joseph Salisbury |
linux (Ubuntu): importance |
Undecided |
Medium |
|
| 2014-03-26 18:34:00 |
Joseph Salisbury |
linux (Ubuntu): status |
Confirmed |
Incomplete |
|
| 2014-03-28 18:34:04 |
Holger Mauermann |
linux (Ubuntu): status |
Incomplete |
Confirmed |
|
| 2014-03-28 18:34:26 |
Holger Mauermann |
tags |
trusty |
kernel-fixed-upstream trusty |
|
| 2014-05-04 00:33:29 |
Michael Kuhn |
bug |
|
|
added subscriber Michael Kuhn |
| 2014-05-17 23:54:33 |
Russell Smith |
bug |
|
|
added subscriber Russell Smith |
| 2014-06-02 09:23:11 |
Colin Sindle |
bug |
|
|
added subscriber Colin Sindle |
| 2014-07-09 20:54:23 |
Fredrik |
bug |
|
|
added subscriber Fredrik |
| 2014-08-05 23:26:38 |
Derek Chen-Becker |
bug |
|
|
added subscriber Derek Chen-Becker |
| 2014-08-13 10:57:25 |
Tim Kuijsten |
bug |
|
|
added subscriber Tim Kuijsten |
| 2014-08-15 12:21:06 |
harm |
bug |
|
|
added subscriber harm |
| 2014-08-26 09:07:56 |
Andreas Ufert |
bug |
|
|
added subscriber Andreas Krause |
| 2014-08-26 21:34:16 |
tobby |
affects |
linux (Ubuntu) |
linux-meta (Ubuntu) |
|
| 2014-08-26 22:00:05 |
Brad Figg |
affects |
linux-meta (Ubuntu) |
linux (Ubuntu) |
|
| 2014-08-26 22:29:47 |
tobby |
attachment added |
|
XML of affected VM https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1297522/+attachment/4187790/+files/fmec.xml |
|
| 2014-08-27 15:50:55 |
Chris J Arges |
nominated for series |
|
Ubuntu Trusty |
|
| 2014-08-27 15:50:55 |
Chris J Arges |
bug task added |
|
linux (Ubuntu Trusty) |
|
| 2014-08-27 15:51:04 |
Chris J Arges |
linux (Ubuntu Trusty): assignee |
|
Chris J Arges (arges) |
|
| 2014-08-27 15:51:12 |
Chris J Arges |
linux (Ubuntu Trusty): status |
New |
In Progress |
|
| 2014-08-27 15:51:16 |
Chris J Arges |
linux (Ubuntu Trusty): importance |
Undecided |
Medium |
|
| 2014-08-28 15:22:49 |
Chris J Arges |
description |
After upgrading some virtual machines (KVM) to Trusty I noticed really high I/O wait times, e.g. Munin graphs now show up to 200 seconds(!) read I/O wait time. See attached image. Of course real latency isn't higher than before, it's only /proc/diskstats that shows totally wrong numbers...
$ cat /proc/diskstats | awk '$3=="vda" { print $7/$4, $11/$8 }'
1375.44 13825.1
From the documentation for /proc/diskstats field 4 is total number of reads completed, field 7 is the total time spent reading in milliseconds, and fields 8 and 11 are the same for writes. So above numbers are the average read and write latency in milliseconds.
Same weird numbers with iowait. Note the column "await" (average time in milliseconds for I/O requests):
$ iostat -dx 1 60
Linux 3.13.0-19-generic (munin) 03/25/14 _x86_64_ (2 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 2.30 16.75 72.45 24.52 572.79 778.37 27.87 1.57 620.00 450.20 1121.83 1.71 16.54
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 52.00 0.00 25.00 0.00 308.00 24.64 0.30 27813.92 0.00 27813.92 0.48 1.20
I upgraded the host system to Trusty too, however there /proc/diskstats output is normal as before.
$ uname -r
3.13.0-19-generic |
SRU Justification:
Impact: Tools that rely on diskstats may report incorrect data in certain conditions. In particular diskstats in a VM may report incorrect statistics.
Fix: 0fec08b4ecfc36fd8a64432343b2964fb86d2675 ( in 3.14-rc1 )
Testcase:
- Install a VM with the affected kernel
- Run cat /proc/diskstats | awk '$3=="vda" { print $7/$4, $11/$8 }'
- If the two values are much larger compared to the v3.14-rc1 kernel in the same VM, we have failed. For example in a failing case I see: "132.44 5458.34"; in a passing case I see: "0.19334 5.90476".
--
After upgrading some virtual machines (KVM) to Trusty I noticed really high I/O wait times, e.g. Munin graphs now show up to 200 seconds(!) read I/O wait time. See attached image. Of course real latency isn't higher than before, it's only /proc/diskstats that shows totally wrong numbers...
$ cat /proc/diskstats | awk '$3=="vda" { print $7/$4, $11/$8 }'
1375.44 13825.1
From the documentation for /proc/diskstats field 4 is total number of reads completed, field 7 is the total time spent reading in milliseconds, and fields 8 and 11 are the same for writes. So above numbers are the average read and write latency in milliseconds.
Same weird numbers with iowait. Note the column "await" (average time in milliseconds for I/O requests):
$ iostat -dx 1 60
Linux 3.13.0-19-generic (munin) 03/25/14 _x86_64_ (2 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 2.30 16.75 72.45 24.52 572.79 778.37 27.87 1.57 620.00 450.20 1121.83 1.71 16.54
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
vda 0.00 52.00 0.00 25.00 0.00 308.00 24.64 0.30 27813.92 0.00 27813.92 0.48 1.20
I upgraded the host system to Trusty too, however there /proc/diskstats output is normal as before.
$ uname -r
3.13.0-19-generic |
|
| 2014-08-28 15:23:49 |
Chris J Arges |
linux (Ubuntu): status |
Confirmed |
Fix Released |
|
| 2014-08-28 18:34:30 |
Tim Gardner |
linux (Ubuntu Trusty): status |
In Progress |
Fix Committed |
|
| 2014-09-04 18:32:53 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/trusty-proposed/linux-keystone |
|
| 2014-09-05 15:09:26 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/precise-proposed/linux-lts-trusty |
|
| 2014-09-08 13:54:03 |
Sergio Oliveira |
bug |
|
|
added subscriber Sergio Oliveira |
| 2014-09-08 20:29:19 |
Brad Figg |
tags |
kernel-fixed-upstream trusty |
kernel-fixed-upstream trusty verification-needed-trusty |
|
| 2014-09-08 22:52:19 |
Holger Mauermann |
tags |
kernel-fixed-upstream trusty verification-needed-trusty |
kernel-fixed-upstream trusty verification-done-trusty |
|
| 2014-09-22 22:45:50 |
Launchpad Janitor |
linux (Ubuntu Trusty): status |
Fix Committed |
Fix Released |
|
| 2014-09-22 22:45:50 |
Launchpad Janitor |
cve linked |
|
2014-3601 |
|
| 2014-09-22 22:45:50 |
Launchpad Janitor |
cve linked |
|
2014-5077 |
|
| 2014-09-22 22:45:50 |
Launchpad Janitor |
cve linked |
|
2014-5472 |
|
| 2014-09-22 23:43:49 |
Launchpad Janitor |
branch linked |
|
lp:ubuntu/precise-security/linux-lts-trusty |
|
