Activity log for bug #1668129

Date Who What changed Old value New value Message
2017-02-27 02:31:46 Pete Cheslock bug added bug
2017-02-27 02:31:46 Pete Cheslock attachment added lspci-vnvn.log https://bugs.launchpad.net/bugs/1668129/+attachment/4827353/+files/lspci-vnvn.log
2017-02-27 03:00:10 Brad Figg linux (Ubuntu): status New Incomplete
2017-02-27 03:04:59 Pete Cheslock tags apport-collected ec2-images xenial
2017-02-27 03:05:01 Pete Cheslock description On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 <snip> [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 <snip> [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Feb 27 02:12 seq crw-rw---- 1 root audio 116, 33 Feb 27 02:12 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A DistroRelease: Ubuntu 16.04 Ec2AMI: ami-bc62b2aa Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: i3.2xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this notice. No journal files were opened due to insufficient permissions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-64-generic root=UUID=cfda0544-9803-41e7-badb-43563085ff3a ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44 RelatedPackageVersions: linux-restricted-modules-4.4.0-64-generic N/A linux-backports-modules-4.4.0-64-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: xenial ec2-images Uname: Linux 4.4.0-64-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: _MarkForUpload: True dmi.bios.date: 12/12/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/12/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen
2017-02-27 03:05:02 Pete Cheslock attachment added CurrentDmesg.txt https://bugs.launchpad.net/bugs/1668129/+attachment/4827354/+files/CurrentDmesg.txt
2017-02-27 03:05:03 Pete Cheslock attachment added Lspci.txt https://bugs.launchpad.net/bugs/1668129/+attachment/4827355/+files/Lspci.txt
2017-02-27 03:05:05 Pete Cheslock attachment added ProcCpuinfo.txt https://bugs.launchpad.net/bugs/1668129/+attachment/4827356/+files/ProcCpuinfo.txt
2017-02-27 03:05:08 Pete Cheslock attachment added ProcInterrupts.txt https://bugs.launchpad.net/bugs/1668129/+attachment/4827357/+files/ProcInterrupts.txt
2017-02-27 03:05:09 Pete Cheslock attachment added ProcModules.txt https://bugs.launchpad.net/bugs/1668129/+attachment/4827358/+files/ProcModules.txt
2017-02-27 03:05:11 Pete Cheslock attachment added UdevDb.txt https://bugs.launchpad.net/bugs/1668129/+attachment/4827359/+files/UdevDb.txt
2017-02-27 03:05:46 Pete Cheslock linux (Ubuntu): status Incomplete Confirmed
2017-02-27 10:02:26 Anne-Marie bug added subscriber Anne-Marie
2017-02-27 18:51:32 Jay Vosburgh bug added subscriber Jay Vosburgh
2017-02-27 22:18:05 Joseph Salisbury linux (Ubuntu): importance Undecided High
2017-02-27 22:18:46 Joseph Salisbury nominated for series Ubuntu Xenial
2017-02-27 22:18:46 Joseph Salisbury bug task added linux (Ubuntu Xenial)
2017-02-27 22:18:52 Joseph Salisbury linux (Ubuntu Xenial): importance Undecided High
2017-02-27 22:18:55 Joseph Salisbury linux (Ubuntu Xenial): status New Triaged
2017-02-27 22:18:59 Joseph Salisbury linux (Ubuntu): status Confirmed Triaged
2017-02-27 22:19:08 Joseph Salisbury tags apport-collected ec2-images xenial apport-collected ec2-images kernel-key xenial
2017-02-27 22:52:21 Joseph Salisbury linux (Ubuntu): importance High Critical
2017-02-27 22:52:24 Joseph Salisbury linux (Ubuntu Xenial): importance High Critical
2017-02-28 00:31:41 Ricky Ramirez bug added subscriber Ricky Ramirez
2017-02-28 08:34:58 Dan Streetman linux (Ubuntu Xenial): assignee Dan Streetman (ddstreet)
2017-02-28 08:35:01 Dan Streetman linux (Ubuntu): assignee Dan Streetman (ddstreet)
2017-03-01 01:42:11 Hao Chen bug added subscriber Hao Chen
2017-03-01 01:57:12 Jacob Scott bug added subscriber Jacob Scott
2017-03-01 19:46:58 Matt Wilson bug added subscriber Matt Wilson
2017-03-01 22:21:27 Kamal Mostafa bug task added linux-aws (Ubuntu)
2017-03-01 22:22:35 Kamal Mostafa linux-aws (Ubuntu): assignee Dan Streetman (ddstreet)
2017-03-01 22:22:51 Kamal Mostafa linux-aws (Ubuntu Xenial): assignee Dan Streetman (ddstreet)
2017-03-01 22:24:31 Kamal Mostafa linux-aws (Ubuntu Xenial): status New Fix Committed
2017-03-01 22:24:35 Kamal Mostafa linux-aws (Ubuntu): status New Triaged
2017-03-01 22:24:38 Kamal Mostafa linux-aws (Ubuntu): importance Undecided Critical
2017-03-01 22:24:40 Kamal Mostafa linux-aws (Ubuntu Xenial): importance Undecided Critical
2017-03-01 22:50:21 Dan Streetman linux-aws (Ubuntu): status Triaged In Progress
2017-03-01 22:50:25 Dan Streetman linux-aws (Ubuntu Xenial): status Fix Committed In Progress
2017-03-02 15:53:45 Dan Streetman linux-aws (Ubuntu): status In Progress Fix Committed
2017-03-02 15:53:47 Dan Streetman linux-aws (Ubuntu Xenial): status In Progress Fix Committed
2017-03-02 20:55:29 Jonathan Camp bug added subscriber Jonathan Camp
2017-03-03 01:12:09 Andrew Lau bug added subscriber Andrew Lau
2017-03-07 14:12:03 José Martínez bug added subscriber José Martínez
2017-03-08 23:51:05 Jeremiah Shirk bug added subscriber Jeremiah Shirk
2017-03-11 16:55:17 Dominique Poulain bug added subscriber Dominique Poulain
2017-03-13 10:16:20 lorello bug added subscriber lorello
2017-03-13 13:28:59 Yotpo bug added subscriber Yotpo
2017-03-15 17:56:15 Joseph Salisbury tags apport-collected ec2-images kernel-key xenial apport-collected ec2-images xenial
2017-03-21 07:56:58 Patrick attachment added kern.log https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668129/+attachment/4841381/+files/kern.log
2017-03-21 07:57:03 Patrick attachment added kern.log https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1668129/+attachment/4841382/+files/kern.log
2017-03-21 14:04:29 Craig Watcham bug added subscriber Craig Watcham
2017-03-21 22:07:52 Patrick description On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 <snip> [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux --- AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Feb 27 02:12 seq crw-rw---- 1 root audio 116, 33 Feb 27 02:12 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A DistroRelease: Ubuntu 16.04 Ec2AMI: ami-bc62b2aa Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: i3.2xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory JournalErrors: Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system. Users in the 'systemd-journal' group can see all messages. Pass -q to turn off this notice. No journal files were opened due to insufficient permissions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=screen-256color PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-64-generic root=UUID=cfda0544-9803-41e7-badb-43563085ff3a ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44 RelatedPackageVersions: linux-restricted-modules-4.4.0-64-generic N/A linux-backports-modules-4.4.0-64-generic N/A linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: xenial ec2-images Uname: Linux 4.4.0-64-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: _MarkForUpload: True dmi.bios.date: 12/12/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/12/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen On the AWS i3 instance class - when putting the new NVME storage disks under high IO load - seeing data corruption and errors in dmesg [ 662.884390] blk_update_request: I/O error, dev nvme0n1, sector 120063912 [ 662.887824] Buffer I/O error on dev nvme0n1, logical block 14971093, lost async page write [ 662.891254] Buffer I/O error on dev nvme0n1, logical block 14971094, lost async page write [ 662.895591] Buffer I/O error on dev nvme0n1, logical block 14971095, lost async page write [ 662.899873] Buffer I/O error on dev nvme0n1, logical block 14971096, lost async page write [ 662.904179] Buffer I/O error on dev nvme0n1, logical block 14971097, lost async page write [ 662.908458] Buffer I/O error on dev nvme0n1, logical block 14971098, lost async page write [ 662.912287] Buffer I/O error on dev nvme0n1, logical block 14971099, lost async page write [ 662.916047] Buffer I/O error on dev nvme0n1, logical block 14971100, lost async page write [ 662.920285] Buffer I/O error on dev nvme0n1, logical block 14971101, lost async page write [ 662.924565] Buffer I/O error on dev nvme0n1, logical block 14971102, lost async page write [ 663.645530] blk_update_request: I/O error, dev nvme0n1, sector 120756912 <snip> [ 1012.752265] blk_update_request: I/O error, dev nvme0n1, sector 3744 [ 1012.755396] buffer_io_error: 194552 callbacks suppressed [ 1012.755398] Buffer I/O error on dev nvme0n1, logical block 20, lost async page write [ 1012.759248] Buffer I/O error on dev nvme0n1, logical block 21, lost async page write [ 1012.763368] Buffer I/O error on dev nvme0n1, logical block 22, lost async page write [ 1012.767271] Buffer I/O error on dev nvme0n1, logical block 23, lost async page write [ 1012.771314] Buffer I/O error on dev nvme0n1, logical block 24, lost async page write Able to replicate this with a bonnie++ stress test. bonnie++ -d /mnt/test/ -r 1000 Linux i-0d76e144d85f487cf 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux --- AlsaDevices:  total 0  crw-rw---- 1 root audio 116, 1 Feb 27 02:12 seq  crw-rw---- 1 root audio 116, 33 Feb 27 02:12 timer AplayDevices: Error: [Errno 2] No such file or directory ApportVersion: 2.20.1-0ubuntu2.5 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: CRDA: N/A DistroRelease: Ubuntu 16.04 Ec2AMI: ami-bc62b2aa Ec2AMIManifest: (unknown) Ec2AvailabilityZone: us-east-1d Ec2InstanceType: i3.2xlarge Ec2Kernel: unavailable Ec2Ramdisk: unavailable IwConfig: Error: [Errno 2] No such file or directory JournalErrors:  Error: command ['journalctl', '-b', '--priority=warning', '--lines=1000'] failed with exit code 1: Hint: You are currently not seeing messages from other users and the system.        Users in the 'systemd-journal' group can see all messages. Pass -q to        turn off this notice.  No journal files were opened due to insufficient permissions. Lsusb: Error: command ['lsusb'] failed with exit code 1: MachineType: Xen HVM domU Package: linux (not installed) PciMultimedia: ProcEnviron:  TERM=screen-256color  PATH=(custom, no user)  XDG_RUNTIME_DIR=<set>  LANG=en_US.UTF-8  SHELL=/bin/bash ProcFB: ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.4.0-64-generic root=UUID=cfda0544-9803-41e7-badb-43563085ff3a ro console=tty1 console=ttyS0 ProcVersionSignature: Ubuntu 4.4.0-64.85-generic 4.4.44 RelatedPackageVersions:  linux-restricted-modules-4.4.0-64-generic N/A  linux-backports-modules-4.4.0-64-generic N/A  linux-firmware N/A RfKill: Error: [Errno 2] No such file or directory Tags: xenial ec2-images Uname: Linux 4.4.0-64-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: WifiSyslog: _MarkForUpload: True dmi.bios.date: 12/12/2016 dmi.bios.vendor: Xen dmi.bios.version: 4.2.amazon dmi.chassis.type: 1 dmi.chassis.vendor: Xen dmi.modalias: dmi:bvnXen:bvr4.2.amazon:bd12/12/2016:svnXen:pnHVMdomU:pvr4.2.amazon:cvnXen:ct1:cvr: dmi.product.name: HVM domU dmi.product.version: 4.2.amazon dmi.sys.vendor: Xen
2017-03-22 15:18:03 Oran Agra bug added subscriber Oran Agra
2017-03-23 01:32:39 Shane Meyers bug added subscriber Shane Meyers
2017-03-29 12:04:17 Amanpreet Singh linux-aws (Ubuntu): status Fix Committed Fix Released
2017-03-29 12:05:10 Amanpreet Singh linux-aws (Ubuntu Xenial): status Fix Committed In Progress
2017-03-29 12:05:17 Amanpreet Singh linux-aws (Ubuntu Xenial): status In Progress Fix Committed
2017-03-29 12:46:58 Dan Streetman linux-aws (Ubuntu): status Fix Released Fix Committed
2017-04-05 09:59:41 Guido Iaquinti bug added subscriber Guido Iaquinti
2017-04-13 00:51:07 Andrew Lau bug task added linux-lts-xenial (Ubuntu)
2017-04-13 00:54:14 Andrew Lau bug task deleted linux-lts-xenial (Ubuntu)
2017-04-13 00:54:30 Andrew Lau bug task deleted linux-lts-xenial (Ubuntu Xenial)
2017-08-04 19:35:29 Dan Streetman linux (Ubuntu Xenial): status Triaged Won't Fix
2017-08-04 19:35:36 Dan Streetman linux (Ubuntu): status Triaged Won't Fix
2017-10-13 15:28:52 Dan Streetman linux-aws (Ubuntu): status Fix Committed Fix Released
2017-10-13 15:28:55 Dan Streetman linux-aws (Ubuntu Xenial): status Fix Committed Fix Released