Amazon I3 Instance Buffer I/O error on dev nvme0n1
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
linux (Ubuntu) |
Won't Fix
|
Critical
|
Dan Streetman | ||
Xenial |
Won't Fix
|
Critical
|
Dan Streetman | ||
linux-aws (Ubuntu) |
Fix Released
|
Critical
|
Dan Streetman | ||
Xenial |
Fix Released
|
Critical
|
Dan Streetman |
Bug Description
On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg
[ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912
[ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write
[ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write
[ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write
[ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write
[ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write
[ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write
[ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write
[ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write
[ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write
[ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write
[ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912
<snip>
[ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744
[ 1012.755396] buffer_io_error: 194552 callbacks suppressed
[ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write
[ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write
[ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write
[ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write
[ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write
Able to replicate this with a bonnie++ stress test.
bonnie++ -d /mnt/test/ -r 1000
Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
---
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Feb 27 02:12 seq
crw-rw---- 1 root audio 116, 33 Feb 27 02:12 timer
AplayDevices: Error: [Errno 2] No such file or directory
ApportVersion: 2.20.1-0ubuntu2.5
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
DistroRelease: Ubuntu 16.04
Ec2AMI: ami-bc62b2aa
Ec2AMIManifest: (unknown)
Ec2Availability
Ec2InstanceType: i3.2xlarge
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
IwConfig: Error: [Errno 2] No such file or directory
JournalErrors:
Error: command ['journalctl', '-b', '--priority=
Users in the 'systemd-journal' group can see all messages. Pass -q to
turn off this notice.
No journal files were opened due to insufficient permissions.
Lsusb: Error: command ['lsusb'] failed with exit code 1:
MachineType: Xen HVM domU
Package: linux (not installed)
PciMultimedia:
ProcEnviron:
TERM=screen-
PATH=(custom, no user)
XDG_RUNTIME_
LANG=en_US.UTF-8
SHELL=/bin/bash
ProcFB:
ProcKernelCmdLine: BOOT_IMAGE=
ProcVersionSign
RelatedPackageV
linux-
linux-
linux-firmware N/A
RfKill: Error: [Errno 2] No such file or directory
Tags: xenial ec2-images
Uname: Linux 4.4.0-64-generic x86_64
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups:
WifiSyslog:
_MarkForUpload: True
dmi.bios.date: 12/12/2016
dmi.bios.vendor: Xen
dmi.bios.version: 4.2.amazon
dmi.chassis.type: 1
dmi.chassis.vendor: Xen
dmi.modalias: dmi:bvnXen:
dmi.product.name: HVM domU
dmi.product.
dmi.sys.vendor: Xen
Changed in linux (Ubuntu): | |
importance: | Undecided → High |
Changed in linux (Ubuntu Xenial): | |
importance: | Undecided → High |
status: | New → Triaged |
Changed in linux (Ubuntu): | |
status: | Confirmed → Triaged |
tags: | added: kernel-key |
Changed in linux (Ubuntu): | |
importance: | High → Critical |
Changed in linux (Ubuntu Xenial): | |
importance: | High → Critical |
Changed in linux (Ubuntu Xenial): | |
assignee: | nobody → Dan Streetman (ddstreet) |
Changed in linux (Ubuntu): | |
assignee: | nobody → Dan Streetman (ddstreet) |
Changed in linux-aws (Ubuntu): | |
assignee: | nobody → Dan Streetman (ddstreet) |
Changed in linux-aws (Ubuntu Xenial): | |
assignee: | nobody → Dan Streetman (ddstreet) |
status: | New → Fix Committed |
Changed in linux-aws (Ubuntu): | |
status: | New → Triaged |
importance: | Undecided → Critical |
Changed in linux-aws (Ubuntu Xenial): | |
importance: | Undecided → Critical |
Changed in linux-aws (Ubuntu): | |
status: | In Progress → Fix Committed |
Changed in linux-aws (Ubuntu Xenial): | |
status: | In Progress → Fix Committed |
tags: | removed: kernel-key |
description: | updated |
Changed in linux-aws (Ubuntu): | |
status: | Fix Released → Fix Committed |
no longer affects: | linux-lts-xenial (Ubuntu) |
no longer affects: | linux-lts-xenial (Ubuntu Xenial) |
Changed in linux-aws (Ubuntu): | |
status: | Fix Committed → Fix Released |
Changed in linux-aws (Ubuntu Xenial): | |
status: | Fix Committed → Fix Released |
tags: | added: kernel-daily-bug |
This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:
apport-collect 1668129
and then change the status of the bug to 'Confirmed'.
If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.
This change has been made by an automated script, maintained by the Ubuntu Kernel Team.