| 2021-06-09 20:37:34 |
Joseph Salisbury |
bug |
|
|
added bug |
| 2021-06-09 20:45:27 |
Joseph Salisbury |
description |
There is a bug in the Linux block layer responsible for merging BIOs that go across the page boundary. This bug was introduced in Linux 5.1 when the block layer BIO page tracking is enhanced to support multiple pages.
Without this patch, data corruption can occur. The change to the kernel block layer in Linux 5.1 changes the way multiple pages are merged to a single block I/O descriptor, and how contiguous block I/O descriptors are merged with previous descriptors.
If contiguous block I/O requests cross a page boundary of 4k, defined by the hv_storvsc driver, the new block merge process can create two pages of block I/O requests (the latter page with an offset) that refer to the same physical sector on disk. This page list is then assembled for the SCSI generic driver.
In the above scenario, when the block I/O request sizes are 512 bytes, the Azure LIS driver (hv_storvsc module) is not able to correctly parse the page array from the SCSI generic driver due to this bug in Linux block layer and creates a potential overflow of offset I/O requests and corruption of data on disk.
Mitigation of data loss is proven with filesystems with block size 4k. When block I/O requests are of sizes 4k or multiples of 4k, they are the page aligned in the memory and are not affected by the block I/O merging algorithm introduced in Linux 5.1. Most modern file systems use 4k I/O block size by default, thus mitigating this problem.
We have submitted a patch to Linux mainline kernel to fix this:
https://patchwork.kernel.org/project/linux-block/patch/1623094445-22332-1-git-send-email-longli@linuxonhyperv.com/
c9c9762d4d44 ("block: return the correct bvec when checking for gaps")
Please include this patch in any supported kernels that are 5.1 or later. |
There is a bug in the Linux block layer responsible for merging BIOs that go across the page boundary. This bug was introduced in Linux 5.1 when the block layer BIO page tracking is enhanced to support multiple pages.
Without this patch, data corruption can occur. The change to the kernel block layer in Linux 5.1 changes the way multiple pages are merged to a single block I/O descriptor, and how contiguous block I/O descriptors are merged with previous descriptors.
If contiguous block I/O requests cross a page boundary of 4k, defined by the hv_storvsc driver, the new block merge process can create two pages of block I/O requests (the latter page with an offset) that refer to the same physical sector on disk. This page list is then assembled for the SCSI generic driver.
In the above scenario, when the block I/O request sizes are 512 bytes, the Azure LIS driver (hv_storvsc module) is not able to correctly parse the page array from the SCSI generic driver due to this bug in Linux block layer and creates a potential overflow of offset I/O requests and corruption of data on disk.
Mitigation of data loss is proven with filesystems with block size 4k. When block I/O requests are of sizes 4k or multiples of 4k, they are the page aligned in the memory and are not affected by the block I/O merging algorithm introduced in Linux 5.1. Most modern file systems use 4k I/O block size by default, thus mitigating this problem.
We have submitted a patch to Linux mainline kernel to fix this:
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-next&id=c9c9762d4d44dcb1b2ba90cfb4122dc11ceebf31
c9c9762d4d44 ("block: return the correct bvec when checking for gaps")
Please include this patch in any supported kernels that are 5.1 or later. |
|
| 2021-06-09 23:14:28 |
Dexuan Cui |
bug |
|
|
added subscriber Dexuan Cui |
| 2021-06-10 11:12:29 |
Tim Gardner |
nominated for series |
|
Ubuntu Impish |
|
| 2021-06-10 11:12:29 |
Tim Gardner |
bug task added |
|
linux-azure (Ubuntu Impish) |
|
| 2021-06-10 11:12:29 |
Tim Gardner |
nominated for series |
|
Ubuntu Focal |
|
| 2021-06-10 11:12:29 |
Tim Gardner |
bug task added |
|
linux-azure (Ubuntu Focal) |
|
| 2021-06-10 11:12:29 |
Tim Gardner |
nominated for series |
|
Ubuntu Hirsute |
|
| 2021-06-10 11:12:29 |
Tim Gardner |
bug task added |
|
linux-azure (Ubuntu Hirsute) |
|
| 2021-06-10 11:12:29 |
Tim Gardner |
nominated for series |
|
Ubuntu Groovy |
|
| 2021-06-10 11:12:29 |
Tim Gardner |
bug task added |
|
linux-azure (Ubuntu Groovy) |
|
| 2021-06-21 15:45:39 |
Tim Gardner |
linux-azure (Ubuntu Focal): status |
New |
In Progress |
|
| 2021-06-21 15:45:44 |
Tim Gardner |
linux-azure (Ubuntu Groovy): status |
New |
In Progress |
|
| 2021-06-21 15:45:48 |
Tim Gardner |
linux-azure (Ubuntu Hirsute): status |
New |
In Progress |
|
| 2021-06-21 15:45:53 |
Tim Gardner |
linux-azure (Ubuntu Groovy): assignee |
|
Tim Gardner (timg-tpi) |
|
| 2021-06-21 15:45:59 |
Tim Gardner |
linux-azure (Ubuntu Hirsute): assignee |
|
Tim Gardner (timg-tpi) |
|
| 2021-06-21 15:46:02 |
Tim Gardner |
linux-azure (Ubuntu Impish): assignee |
|
Tim Gardner (timg-tpi) |
|
| 2021-06-21 15:48:29 |
Tim Gardner |
tags |
|
bot-stop-nagging |
|
| 2021-06-22 06:48:13 |
Stefan Bader |
linux-azure (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2021-06-22 06:48:17 |
Stefan Bader |
linux-azure (Ubuntu Groovy): importance |
Undecided |
Medium |
|
| 2021-06-22 06:48:21 |
Stefan Bader |
linux-azure (Ubuntu Hirsute): importance |
Undecided |
Medium |
|
| 2021-06-23 12:00:28 |
Tim Gardner |
linux-azure (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2021-06-23 12:11:52 |
Tim Gardner |
linux-azure (Ubuntu Groovy): status |
In Progress |
Fix Committed |
|
| 2021-06-23 12:29:47 |
Tim Gardner |
linux-azure (Ubuntu Hirsute): status |
In Progress |
Fix Committed |
|
| 2021-06-25 10:28:05 |
Ubuntu Kernel Bot |
tags |
bot-stop-nagging |
bot-stop-nagging verification-needed-groovy |
|
| 2021-06-26 07:03:22 |
Ubuntu Kernel Bot |
tags |
bot-stop-nagging verification-needed-groovy |
bot-stop-nagging verification-needed-groovy verification-needed-hirsute |
|
| 2021-07-06 17:08:10 |
Tim Gardner |
description |
There is a bug in the Linux block layer responsible for merging BIOs that go across the page boundary. This bug was introduced in Linux 5.1 when the block layer BIO page tracking is enhanced to support multiple pages.
Without this patch, data corruption can occur. The change to the kernel block layer in Linux 5.1 changes the way multiple pages are merged to a single block I/O descriptor, and how contiguous block I/O descriptors are merged with previous descriptors.
If contiguous block I/O requests cross a page boundary of 4k, defined by the hv_storvsc driver, the new block merge process can create two pages of block I/O requests (the latter page with an offset) that refer to the same physical sector on disk. This page list is then assembled for the SCSI generic driver.
In the above scenario, when the block I/O request sizes are 512 bytes, the Azure LIS driver (hv_storvsc module) is not able to correctly parse the page array from the SCSI generic driver due to this bug in Linux block layer and creates a potential overflow of offset I/O requests and corruption of data on disk.
Mitigation of data loss is proven with filesystems with block size 4k. When block I/O requests are of sizes 4k or multiples of 4k, they are the page aligned in the memory and are not affected by the block I/O merging algorithm introduced in Linux 5.1. Most modern file systems use 4k I/O block size by default, thus mitigating this problem.
We have submitted a patch to Linux mainline kernel to fix this:
https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git/commit/?h=for-next&id=c9c9762d4d44dcb1b2ba90cfb4122dc11ceebf31
c9c9762d4d44 ("block: return the correct bvec when checking for gaps")
Please include this patch in any supported kernels that are 5.1 or later. |
[Impact]
There is a bug in the Linux block layer responsible for merging BIOs that go across the page boundary. This bug was introduced in Linux 5.1 when the block layer BIO page tracking is enhanced to support multiple pages.
Without this patch, data corruption can occur. The change to the kernel block layer in Linux 5.1 changes the way multiple pages are merged to a single block I/O descriptor, and how contiguous block I/O descriptors are merged with previous descriptors.
If contiguous block I/O requests cross a page boundary of 4k, defined by the hv_storvsc driver, the new block merge process can create two pages of block I/O requests (the latter page with an offset) that refer to the same physical sector on disk. This page list is then assembled for the SCSI generic driver.
In the above scenario, when the block I/O request sizes are 512 bytes, the Azure LIS driver (hv_storvsc module) is not able to correctly parse the page array from the SCSI generic driver due to this bug in Linux block layer and creates a potential overflow of offset I/O requests and corruption of data on disk.
Mitigation of data loss is proven with filesystems with block size 4k. When block I/O requests are of sizes 4k or multiples of 4k, they are the page aligned in the memory and are not affected by the block I/O merging algorithm introduced in Linux 5.1. Most modern file systems use 4k I/O block size by default, thus mitigating this problem.
An upstream patch fixes this bug: commit c9c9762d4d44dcb1b2ba90cfb4122dc11ceebf31 ("block: return the correct bvec when checking for gaps")
Please include this patch in any supported kernels that are 5.1 or later.
[Test Plan]
stress-ng --sequential 8 --class io -t 5m --times
[Where problems could occur]
Different incorrect pages could be wriiten to disk.
[Other Info]
This patch has already been released in all [FGHI] Azure kernels. |
|
| 2021-07-06 17:08:24 |
Tim Gardner |
bug task added |
|
linux (Ubuntu) |
|
| 2021-07-06 17:08:41 |
Tim Gardner |
linux (Ubuntu Focal): importance |
Undecided |
Medium |
|
| 2021-07-06 17:08:41 |
Tim Gardner |
linux (Ubuntu Focal): status |
New |
In Progress |
|
| 2021-07-06 17:09:00 |
Tim Gardner |
linux (Ubuntu Groovy): importance |
Undecided |
Medium |
|
| 2021-07-06 17:09:00 |
Tim Gardner |
linux (Ubuntu Groovy): status |
New |
In Progress |
|
| 2021-07-06 17:09:16 |
Tim Gardner |
linux (Ubuntu Hirsute): importance |
Undecided |
Medium |
|
| 2021-07-06 17:09:16 |
Tim Gardner |
linux (Ubuntu Hirsute): status |
New |
In Progress |
|
| 2021-07-06 17:09:31 |
Tim Gardner |
linux (Ubuntu Impish): importance |
Undecided |
Medium |
|
| 2021-07-06 17:09:31 |
Tim Gardner |
linux (Ubuntu Impish): status |
New |
In Progress |
|
| 2021-07-12 09:56:47 |
Ubuntu Kernel Bot |
tags |
bot-stop-nagging verification-needed-groovy verification-needed-hirsute |
bot-stop-nagging verification-needed-focal verification-needed-groovy verification-needed-hirsute |
|
| 2021-07-19 15:34:14 |
Tim Gardner |
tags |
bot-stop-nagging verification-needed-focal verification-needed-groovy verification-needed-hirsute |
bot-stop-nagging verification-done-focal verification-done-groovy verification-done-hirsute |
|
| 2021-07-20 16:19:15 |
Launchpad Janitor |
linux-azure (Ubuntu Hirsute): status |
Fix Committed |
Fix Released |
|
| 2021-07-20 16:19:15 |
Launchpad Janitor |
cve linked |
|
2021-33909 |
|
| 2021-07-20 16:21:17 |
Launchpad Janitor |
linux-azure (Ubuntu Groovy): status |
Fix Committed |
Fix Released |
|
| 2021-07-20 16:23:31 |
Launchpad Janitor |
linux-azure (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-07-29 12:13:13 |
Tim Gardner |
linux (Ubuntu Focal): status |
In Progress |
Fix Committed |
|
| 2021-07-29 12:13:27 |
Tim Gardner |
linux (Ubuntu Groovy): status |
In Progress |
Won't Fix |
|
| 2021-07-29 12:13:42 |
Tim Gardner |
linux (Ubuntu Hirsute): status |
In Progress |
Fix Committed |
|
| 2021-07-29 12:13:55 |
Tim Gardner |
linux (Ubuntu Impish): status |
In Progress |
Fix Released |
|
| 2021-07-29 12:14:10 |
Tim Gardner |
linux (Ubuntu Focal): assignee |
|
Tim Gardner (timg-tpi) |
|
| 2021-07-29 12:14:23 |
Tim Gardner |
linux (Ubuntu Hirsute): assignee |
|
Tim Gardner (timg-tpi) |
|
| 2021-08-18 23:09:30 |
Launchpad Janitor |
linux-azure (Ubuntu Impish): status |
New |
Fix Released |
|
| 2021-09-07 13:48:28 |
Launchpad Janitor |
linux (Ubuntu Focal): status |
Fix Committed |
Fix Released |
|
| 2021-09-07 13:48:28 |
Launchpad Janitor |
cve linked |
|
2021-3653 |
|
| 2021-09-07 13:48:28 |
Launchpad Janitor |
cve linked |
|
2021-3656 |
|
| 2021-09-07 13:53:28 |
Launchpad Janitor |
linux (Ubuntu Hirsute): status |
Fix Committed |
Fix Released |
|
| 2021-09-07 13:53:28 |
Launchpad Janitor |
cve linked |
|
2020-26541 |
|
