This bug appeared to surface on aws Bionic linux-aws kernel-version": "4.15.0-1035.37", AMD64 We also see this occurring on arm64 instances a1.medium so it does not appear to be arch specific. AMD64 Instances: c5.large , m4.large, m5a.large, r3.large , r5.large, t2.small and t3.medium. This is not 100% reproducible, if affects different instance types during each SRU cycle and appears to be random. However the failure, although not identical bit for bit, is a buffer I/O error I restarted a select few of the failed tests some passed completely and some failed again. I attempted to reproduce on m5a.large. 1.) Deploy m5a.large using cloud prov script with Bionic, ensure kernel is 4.15.0-1035.37 (in proposed at the time of writing this comment) 2.) Clone autotest and autotest-client-tests: git clone https://github.com/autotest/autotest.git git clone git://kernel.ubuntu.com/ubuntu/autotest-client-tests 3.) sudo apt-get update && sudo apt-get install python-pkg-resources 4.) untu@ip-172-31-10-8:~/autotest/client$ sudo ./autotest-local -d ~/autotest-client-tests/ ~/autotest-client-tests/ubuntu_nbd_smoke_test/control 19:42:55 INFO | Writing results to /home/ubuntu/autotest/client/results/default 19:42:55 INFO | START ---- ---- timestamp=1553283775 localtime=Mar 22 19:42:55 19:42:55 INFO | START ubuntu_nbd_smoke_test.nbd-smoke-test ubuntu_nbd_smoke_test.nbd-smoke-test timestamp=1553283775 timeout=600 localtime=Mar 22 19:42:55 19:42:58 ERROR| [stderr] bs=1024, sz=134217728 bytes 19:42:58 ERROR| [stderr] timeout=30 19:42:58 ERROR| [stderr] e2fsck 1.44.1 (24-Mar-2018) 19:43:01 ERROR| Exception escaping from test: Traceback (most recent call last): File "/home/ubuntu/autotest/client/shared/test.py", line 411, in _exec _call_test_function(self.execute, *p_args, **p_dargs) File "/home/ubuntu/autotest/client/shared/test.py", line 823, in _call_test_function return func(*args, **dargs) File "/home/ubuntu/autotest/client/shared/test.py", line 291, in execute postprocess_profiled_run, args, dargs) File "/home/ubuntu/autotest/client/shared/test.py", line 212, in _call_run_once self.run_once(*args, **dargs) File "/home/ubuntu/autotest-client-tests/ubuntu_nbd_smoke_test/ubuntu_nbd_smoke_test.py", line 51, in run_once self.results = utils.system_output(cmd, retain_output=True) File "/home/ubuntu/autotest/client/shared/utils.py", line 1267, in system_output verbose=verbose, args=args).stdout File "/home/ubuntu/autotest/client/shared/utils.py", line 918, in run "Command returned non-zero exit status") CmdError: Command failed, rc=1, Command returned non-zero exit status * Command: /home/ubuntu/autotest-client- tests/ubuntu_nbd_smoke_test/ubuntu_nbd_smoke_test.sh Exit status: 1 Duration: 6.14667606354 stdout: creating backing nbd image /tmp/nbd_image.img -------------------------------------------------------------------------------- Image path: /tmp/nbd_image.img Mount point: /mnt/nbd-test-7736 Date: Fri Mar 22 19:42:56 UTC 2019 Host: ip-172-31-10-8 Kernel: 4.15.0-1034-aws #36-Ubuntu SMP Tue Mar 5 23:17:16 UTC 2019 Machine: ip-172-31-10-8 x86_64 x86_64 CPUs online: 2 CPUs total: 2 Page size: 4096 Pages avail: 1641331 Pages total: 1977692 Free space: Filesystem Size Used Avail Use% Mounted on udev 3.8G 0 3.8G 0% /dev tmpfs 773M 800K 772M 1% /run /dev/nvme0n1p1 7.7G 1.8G 6.0G 23% / tmpfs 3.8G 0 3.8G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup /dev/loop0 92M 92M 0 100% /snap/core/6531 /dev/loop1 18M 18M 0 100% /snap/amazon-ssm-agent/1068 tmpfs 773M 0 773M 0% /run/user/1000 -------------------------------------------------------------------------------- NBD device /dev/nbd0 created found nbd export NBD exports found: test starting client with NBD device /dev/nbd0 Negotiation: ..size = 128MB creating ext4 on /dev/nbd0 mkfs on /dev/nbd0 succeeded after 0 attempt(s) checking ext4 on /dev/nbd0 fsck from util-linux 2.31.1 /dev/nbd0: clean, 11/32768 files, 9787/131072 blocks mount: /dev/nbd0 on /mnt/nbd-test-7736 type ext4 (rw,relatime,data=ordered) mounted on /dev/nbd0 free: Filesystem 1K-blocks Used Available Use% Mounted on /dev/nbd0 122835 1550 112111 2% /mnt/nbd-test-7736 creating large file /mnt/nbd-test-7736/largefile -rw-r--r-- 1 root root 100M Mar 22 19:42 /mnt/nbd-test-7736/largefile free: Filesystem 1K-blocks Used Available Use% Mounted on /dev/nbd0 122835 103951 9710 92% /mnt/nbd-test-7736 removing file /mnt/nbd-test-7736/largefile unmounting /mnt/nbd-test-7736 stopping client disconnect, sock, done Found kernel warning, IO error and/or call trace echo [ 703.195406] creating backing nbd image /tmp/nbd_image.img [ 704.101923] NBD device /dev/nbd0 created [ 704.650458] found nbd export [ 705.698030] starting client with NBD device /dev/nbd0 [ 705.702106] creating ext4 on /dev/nbd0 [ 706.093950] mkfs on /dev/nbd0 succeeded after 0 attempt(s) [ 706.128680] checking ext4 on /dev/nbd0 [ 706.141553] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 706.151522] mounted on /dev/nbd0 [ 706.153205] creating large file /mnt/nbd-test-7736/largefile [ 706.645699] removing file /mnt/nbd-test-7736/largefile [ 706.650698] unmounting /mnt/nbd-test-7736 [ 707.695006] stopping client [ 707.696875] block nbd0: NBD_DISCONNECT [ 707.696933] block nbd0: shutting down sockets [ 707.697399] nbd0: detected capacity change from 0 to 134217728 [ 707.697426] print_req_error: I/O error, dev nbd0, sector 0 [ 707.699986] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.703114] print_req_error: I/O error, dev nbd0, sector 0 [ 707.705603] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.708843] print_req_error: I/O error, dev nbd0, sector 0 [ 707.711471] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.714714] ldm_validate_partition_table(): Disk read failed. [ 707.714727] print_req_error: I/O error, dev nbd0, sector 0 [ 707.717407] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.720647] print_req_error: I/O error, dev nbd0, sector 0 [ 707.723258] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.726546] print_req_error: I/O error, dev nbd0, sector 0 [ 707.729177] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.732395] print_req_error: I/O error, dev nbd0, sector 0 [ 707.735010] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.738272] Dev nbd0: unable to read RDB block 0 [ 707.740565] print_req_error: I/O error, dev nbd0, sector 0 [ 707.743173] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.746462] print_req_error: I/O error, dev nbd0, sector 24 [ 707.749144] Buffer I/O error on dev nbd0, logical block 3, async page read [ 707.752389] print_req_error: I/O error, dev nbd0, sector 0 [ 707.755007] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.759594] nbd0: unable to read partition table [ 708.163286] Found kernel warning, IO error and/or call trace [ 708.163332] echo killing server ================================================================================ Completed Kernel issues: Found kernel warning, IO error and/or call trace: TEST: [ 703.195406] creating backing nbd image /tmp/nbd_image.img [ 704.101923] NBD device /dev/nbd0 created [ 704.650458] found nbd export [ 705.698030] starting client with NBD device /dev/nbd0 [ 705.702106] creating ext4 on /dev/nbd0 [ 706.093950] mkfs on /dev/nbd0 succeeded after 0 attempt(s) [ 706.128680] checking ext4 on /dev/nbd0 [ 706.141553] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 706.151522] mounted on /dev/nbd0 [ 706.153205] creating large file /mnt/nbd-test-7736/largefile [ 706.645699] removing file /mnt/nbd-test-7736/largefile [ 706.650698] unmounting /mnt/nbd-test-7736 [ 707.695006] stopping client [ 707.696875] block nbd0: NBD_DISCONNECT [ 707.696933] block nbd0: shutting down sockets [ 707.697399] nbd0: detected capacity change from 0 to 134217728 [ 707.697426] print_req_error: I/O error, dev nbd0, sector 0 [ 707.699986] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.703114] print_req_error: I/O error, dev nbd0, sector 0 [ 707.705603] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.708843] print_req_error: I/O error, dev nbd0, sector 0 [ 707.711471] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.714714] ldm_validate_partition_table(): Disk read failed. [ 707.714727] print_req_error: I/O error, dev nbd0, sector 0 [ 707.717407] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.720647] print_req_error: I/O error, dev nbd0, sector 0 [ 707.723258] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.726546] print_req_error: I/O error, dev nbd0, sector 0 [ 707.729177] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.732395] print_req_error: I/O error, dev nbd0, sector 0 [ 707.735010] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.738272] Dev nbd0: unable to read RDB block 0 [ 707.740565] print_req_error: I/O error, dev nbd0, sector 0 [ 707.743173] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.746462] print_req_error: I/O error, dev nbd0, sector 24 [ 707.749144] Buffer I/O error on dev nbd0, logical block 3, async page read [ 707.752389] print_req_error: I/O error, dev nbd0, sector 0 [ 707.755007] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.759594] nbd0: unable to read partition table [ 708.163286] Found kernel warning, IO error and/or call trace [ 708.163332] echo stderr: bs=1024, sz=134217728 bytes timeout=30 e2fsck 1.44.1 (24-Mar-2018) 19:43:01 ERROR| child process failed 19:43:03 INFO | ERROR ubuntu_nbd_smoke_test.nbd-smoke-test ubuntu_nbd_smoke_test.nbd-smoke-test timestamp=1553283783 localtime=Mar 22 19:43:03 Command failed, rc=1, Command returned non-zero exit status * Command: /home/ubuntu/autotest-client- tests/ubuntu_nbd_smoke_test/ubuntu_nbd_smoke_test.sh Exit status: 1 Duration: 6.14667606354 stdout: creating backing nbd image /tmp/nbd_image.img -------------------------------------------------------------------------------- Image path: /tmp/nbd_image.img Mount point: /mnt/nbd-test-7736 Date: Fri Mar 22 19:42:56 UTC 2019 Host: ip-172-31-10-8 Kernel: 4.15.0-1034-aws #36-Ubuntu SMP Tue Mar 5 23:17:16 UTC 2019 Machine: ip-172-31-10-8 x86_64 x86_64 CPUs online: 2 CPUs total: 2 Page size: 4096 Pages avail: 1641331 Pages total: 1977692 Free space: Filesystem Size Used Avail Use% Mounted on udev 3.8G 0 3.8G 0% /dev tmpfs 773M 800K 772M 1% /run /dev/nvme0n1p1 7.7G 1.8G 6.0G 23% / tmpfs 3.8G 0 3.8G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 3.8G 0 3.8G 0% /sys/fs/cgroup /dev/loop0 92M 92M 0 100% /snap/core/6531 /dev/loop1 18M 18M 0 100% /snap/amazon-ssm-agent/1068 tmpfs 773M 0 773M 0% /run/user/1000 -------------------------------------------------------------------------------- NBD device /dev/nbd0 created found nbd export NBD exports found: test starting client with NBD device /dev/nbd0 Negotiation: ..size = 128MB creating ext4 on /dev/nbd0 mkfs on /dev/nbd0 succeeded after 0 attempt(s) checking ext4 on /dev/nbd0 fsck from util-linux 2.31.1 /dev/nbd0: clean, 11/32768 files, 9787/131072 blocks mount: /dev/nbd0 on /mnt/nbd-test-7736 type ext4 (rw,relatime,data=ordered) mounted on /dev/nbd0 free: Filesystem 1K-blocks Used Available Use% Mounted on /dev/nbd0 122835 1550 112111 2% /mnt/nbd-test-7736 creating large file /mnt/nbd-test-7736/largefile -rw-r--r-- 1 root root 100M Mar 22 19:42 /mnt/nbd-test-7736/largefile free: Filesystem 1K-blocks Used Available Use% Mounted on /dev/nbd0 122835 103951 9710 92% /mnt/nbd-test-7736 removing file /mnt/nbd-test-7736/largefile unmounting /mnt/nbd-test-7736 stopping client disconnect, sock, done Found kernel warning, IO error and/or call trace echo [ 703.195406] creating backing nbd image /tmp/nbd_image.img [ 704.101923] NBD device /dev/nbd0 created [ 704.650458] found nbd export [ 705.698030] starting client with NBD device /dev/nbd0 [ 705.702106] creating ext4 on /dev/nbd0 [ 706.093950] mkfs on /dev/nbd0 succeeded after 0 attempt(s) [ 706.128680] checking ext4 on /dev/nbd0 [ 706.141553] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 706.151522] mounted on /dev/nbd0 [ 706.153205] creating large file /mnt/nbd-test-7736/largefile [ 706.645699] removing file /mnt/nbd-test-7736/largefile [ 706.650698] unmounting /mnt/nbd-test-7736 [ 707.695006] stopping client [ 707.696875] block nbd0: NBD_DISCONNECT [ 707.696933] block nbd0: shutting down sockets [ 707.697399] nbd0: detected capacity change from 0 to 134217728 [ 707.697426] print_req_error: I/O error, dev nbd0, sector 0 [ 707.699986] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.703114] print_req_error: I/O error, dev nbd0, sector 0 [ 707.705603] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.708843] print_req_error: I/O error, dev nbd0, sector 0 [ 707.711471] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.714714] ldm_validate_partition_table(): Disk read failed. [ 707.714727] print_req_error: I/O error, dev nbd0, sector 0 [ 707.717407] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.720647] print_req_error: I/O error, dev nbd0, sector 0 [ 707.723258] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.726546] print_req_error: I/O error, dev nbd0, sector 0 [ 707.729177] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.732395] print_req_error: I/O error, dev nbd0, sector 0 [ 707.735010] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.738272] Dev nbd0: unable to read RDB block 0 [ 707.740565] print_req_error: I/O error, dev nbd0, sector 0 [ 707.743173] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.746462] print_req_error: I/O error, dev nbd0, sector 24 [ 707.749144] Buffer I/O error on dev nbd0, logical block 3, async page read [ 707.752389] print_req_error: I/O error, dev nbd0, sector 0 [ 707.755007] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.759594] nbd0: unable to read partition table [ 708.163286] Found kernel warning, IO error and/or call trace [ 708.163332] echo killing server ================================================================================ Completed Kernel issues: Found kernel warning, IO error and/or call trace: TEST: [ 703.195406] creating backing nbd image /tmp/nbd_image.img [ 704.101923] NBD device /dev/nbd0 created [ 704.650458] found nbd export [ 705.698030] starting client with NBD device /dev/nbd0 [ 705.702106] creating ext4 on /dev/nbd0 [ 706.093950] mkfs on /dev/nbd0 succeeded after 0 attempt(s) [ 706.128680] checking ext4 on /dev/nbd0 [ 706.141553] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ 706.151522] mounted on /dev/nbd0 [ 706.153205] creating large file /mnt/nbd-test-7736/largefile [ 706.645699] removing file /mnt/nbd-test-7736/largefile [ 706.650698] unmounting /mnt/nbd-test-7736 [ 707.695006] stopping client [ 707.696875] block nbd0: NBD_DISCONNECT [ 707.696933] block nbd0: shutting down sockets [ 707.697399] nbd0: detected capacity change from 0 to 134217728 [ 707.697426] print_req_error: I/O error, dev nbd0, sector 0 [ 707.699986] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.703114] print_req_error: I/O error, dev nbd0, sector 0 [ 707.705603] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.708843] print_req_error: I/O error, dev nbd0, sector 0 [ 707.711471] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.714714] ldm_validate_partition_table(): Disk read failed. [ 707.714727] print_req_error: I/O error, dev nbd0, sector 0 [ 707.717407] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.720647] print_req_error: I/O error, dev nbd0, sector 0 [ 707.723258] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.726546] print_req_error: I/O error, dev nbd0, sector 0 [ 707.729177] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.732395] print_req_error: I/O error, dev nbd0, sector 0 [ 707.735010] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.738272] Dev nbd0: unable to read RDB block 0 [ 707.740565] print_req_error: I/O error, dev nbd0, sector 0 [ 707.743173] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.746462] print_req_error: I/O error, dev nbd0, sector 24 [ 707.749144] Buffer I/O error on dev nbd0, logical block 3, async page read [ 707.752389] print_req_error: I/O error, dev nbd0, sector 0 [ 707.755007] Buffer I/O error on dev nbd0, logical block 0, async page read [ 707.759594] nbd0: unable to read partition table [ 708.163286] Found kernel warning, IO error and/or call trace [ 708.163332] echo stderr: bs=1024, sz=134217728 bytes timeout=30 e2fsck 1.44.1 (24-Mar-2018) 19:43:03 INFO | END ERROR ubuntu_nbd_smoke_test.nbd-smoke-test ubuntu_nbd_smoke_test.nbd-smoke-test timestamp=1553283783 localtime=Mar 22 19:43:03 19:43:03 INFO | END GOOD ---- ---- timestamp=1553283783 localtime=Mar 22 19:43:03 19:43:03 INFO | Report successfully generated at /home/ubuntu/autotest/client/results/default/job_report.html