Bionic: btrfs: kernel BUG at /build/linux-eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233!
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| | Ubuntu on IBM z Systems |
Undecided
|
Unassigned | ||
| | linux (Ubuntu) |
Medium
|
Mauricio Faria de Oliveira | ||
| | Bionic |
High
|
Mauricio Faria de Oliveira | ||
| | Focal |
Medium
|
Mauricio Faria de Oliveira | ||
| | Groovy |
Medium
|
Mauricio Faria de Oliveira | ||
Bug Description
[Impact]
* Users of btrfs started hitting a kernel BUG() (below)
after upgrade from 4.15.0-99.100 to 4.15.0-109.110,
which has 55 btrfs changes.
kernel BUG at /build/
...
Krnl PSW : 00000000be9cb874 00000000ef3786e8 (btrfs_
...
[...] Call Trace:
[...] btrfs_set_
[...] __btrfs_
[...] btrfs_log_
[...] btrfs_log_
[...] btrfs_log_
[...] btrfs_log_
[...] btrfs_sync_
[...] do_fsync+0x5e/0x90
[...] SyS_fdatasync+
[...] system_
$ git log --oneline Ubuntu-
55
* The error happens at random moments, regardless of a
particular activity/load. Workaround is to downgrade.
[Fix]
* This BUG()/function is addressed in patch 4/4 [1] of series
'btrfs: Enhanced runtime defence against fuzzed images' [2],
after issues in the real world, not just crafted fs images:
'one internal report has hit one BUG_ON() with real world fs'
kernel BUG at fs/btrfs/
...
RIP: 0010:btrfs_
* The patch/set [3] is applied in v5.10-rc1 and Ubuntu Unstable:
- d16c702fe4f2 btrfs: ctree: check key order before merging tree blocks
- 07cce5cf3b48 btrfs: extent-tree: kill the BUG_ON() in insert_
- 1c2a07f598d5 btrfs: extent-tree: kill BUG_ON() in __btrfs_
- f98b6215d7d1 btrfs: extent_io: do extra check for extent buffer read write functions
[Test Case]
* There is working synthetic reproducer for this issue,
which is hard to reproduce as reported in commit [4]
that introduces debugging for the issue.
* Regression tests with xfstests and stress-ng shows
no regressions between un/patched kernels.
[Other Info]
* Trivial backports (only refreshing a few context lines)
with 3 more dependency patches on Bionic and 1 on Focal.
And Bionic needed one extra hunk to '#include' a header.
Groovy all apply cleanly.
[1] https://<email address hidden>/
[2] https://<email address hidden>/
[3] https:/
[4] https:/
CVE References
| Mauricio Faria de Oliveira (mfo) wrote : | #1 |
| Changed in linux (Ubuntu): | |
| status: | New → Fix Committed |
| importance: | Undecided → Medium |
| assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
| Changed in linux (Ubuntu Bionic): | |
| status: | New → In Progress |
| importance: | Undecided → High |
| assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
| Changed in linux (Ubuntu Focal): | |
| status: | New → In Progress |
| importance: | Undecided → Medium |
| assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
| Changed in linux (Ubuntu Groovy): | |
| status: | New → In Progress |
| importance: | Undecided → Medium |
| assignee: | nobody → Mauricio Faria de Oliveira (mfo) |
| tags: | added: sts |
| Mauricio Faria de Oliveira (mfo) wrote : | #2 |
Regression testing done with 3 approaches:
1) synthetic reproducer in kernel.org BZ#202833 [1]
2) stress-ng --class filesystem,io
3) xfstests (aka fstests)
No regressions observed in Bionic/Focal/Groovy between original/patched kernels, in s390x native VMs in canonistack-bos01.
| Mauricio Faria de Oliveira (mfo) wrote : | #3 |
1) synthetic reproducer
The reproducer consists in a crafted disk/btrfs image and C test-case.
This has not been helpful.
In Bionic, the crafted disk/btrfs image does not even mount.
In Focal/Groovy it does mount, but the test-case does not trigger any errors.
Anyway, the behavior is the same in the original/patched kernels.
No regressions.
| Mauricio Faria de Oliveira (mfo) wrote : | #4 |
2) stress-ng --class filesystem,io
Compared the stress-ng output and kernel log between original/patched kernels.
No regressions.
Actually, in Bionic the original kernel didn't even finish stress-ng in ~20 hours, while the patched kernel finished it in ~1.5 hours (in line w/ other the releases.)
Thus it's apparently an improvement.
For documentation purposes, this seems to be due to the 'fiemap' stressor in the 'filesystem' class, which had defunct processes when I checked before cancelling the tests:
$ ps axf
...
2659 ? Ss 0:00 tmux new
2661 pts/2 Ss 0:00 \_ -bash
2692 pts/2 S+ 0:00 | \_ /bin/sh ./run-stressng.sh
2743 pts/2 S+ 0:00 | \_ sudo stress-ng --sequential 2 --class filesystem,io -t 2m
2745 pts/2 SL+ 0:00 | \_ stress-ng --sequential 2 --class filesystem io -t 2m
2852 pts/2 D+ 0:00 | \_ stress-ng --sequential 2 --class filesystem io -t 2m
2854 pts/2 D+ 0:01 | | \_ stress-ng --sequential 2 --class filesystem io -t 2m
2856 pts/2 D+ 0:01 | | \_ stress-ng --sequential 2 --class filesystem io -t 2m
2859 pts/2 D+ 0:01 | | \_ stress-ng --sequential 2 --class filesystem io -t 2m
2860 pts/2 D+ 0:01 | | \_ stress-ng --sequential 2 --class filesystem io -t 2m
2853 pts/2 D+ 0:00 | \_ stress-ng --sequential 2 --class filesystem io -t 2m
2855 pts/2 Z+ 322:29 | \_ [stress-ng-fiema] <defunct>
2857 pts/2 Z+ 323:55 | \_ [stress-ng-fiema] <defunct>
2858 pts/2 Z+ 321:50 | \_ [stress-ng-fiema] <defunct>
2861 pts/2 Z+ 322:51 | \_ [stress-ng-fiema] <defunct>
11823 pts/4 Ss 0:00 \_ -bash
...
| Mauricio Faria de Oliveira (mfo) wrote : | #5 |
[1] kernel.org BZ#202833
https:/
| Mauricio Faria de Oliveira (mfo) wrote : | #6 |
3) xfstests (aka fstests)
Compared the 'consistently fail set' and 'likelyhood to fail' in flaky tests (see below.)
No regressions.
Details:
---
Source:
- git://git.
- commit 31f6949f ("ext4: verify unwritten extent conversion in buff-io")
The test set is the 'full' suite (default), not just the 'quick' group/suite.
The fail set is flaky (some tests fail/pass inconsistently), so the 'consistently fail set' (i.e., fail every time) across 10-20 runs has been compared between the original/patched kernels.
Also looked for increased likelyhood to fail (i.e., flaky tests that fail more often with the patches.)
More Details:
---
The xfstests output/log includes the fail set in the 'Failures:' line, and the kernel version in the 'PLATFORM' line.
For example:
$ grep -e ^Fail -e ^PLATFORM xfstests.
PLATFORM -- Linux/s390x mfo-s390x-focal 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:52:40 UTC 2020
Failures: btrfs/153 btrfs/200 btrfs/204 btrfs/205 btrfs/213 btrfs/220 btrfs/221 btrfs/222 btrfs/226 generic/166 generic/175 generic/260 generic/286 generic/301 generic/465 generic/484 generic/610
Those two lines from each log/run are used to identify, per kernel version (original/patched):
- how many runs have been done;
- which tests fail consistently (i.e., in all runs);
- which tests fail inconsistently (i.e., flaky) and how many times (to compare likelyhood to fail)
| Mauricio Faria de Oliveira (mfo) wrote : | #7 |
Bionic/xfstests
======
Original/patched kernel versions:
$ grep -h ^PLATFORM xfstests.log.* | sort -u
PLATFORM -- Linux/s390x mfo-s390x-bionic 4.15.0-122-generic #124+test20201026b1 SMP Mon Oct 26 13:10:37 -03 2020
PLATFORM -- Linux/s390x mfo-s390x-bionic 4.15.0-122-generic #124-Ubuntu SMP Thu Oct 15 13:02:05 UTC 2020
Original kernel:
---
$ VERSION=
Number of runs:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
10
$ RUNS=10
Consistent fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
10 btrfs/186
10 btrfs/193
10 btrfs/200
10 btrfs/202
10 btrfs/204
10 btrfs/205
10 btrfs/214
10 btrfs/219
10 btrfs/220
10 btrfs/221
10 btrfs/222
10 generic/260
10 generic/286
10 generic/301
10 generic/371
10 generic/405
10 generic/407
10 generic/465
10 generic/480
10 generic/488
10 generic/502
10 generic/513
10 generic/526
10 generic/527
10 generic/546
10 generic/562
10 generic/565
Flaky fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
1 btrfs/028
1 generic/089
1 generic/464
2 generic/175
3 generic/476
4 generic/531
7 btrfs/078
8 btrfs/010
8 generic/074
8 generic/142
8 generic/143
9 generic/133
9 generic/475
Patched kernel:
---
$ VERSION=
Number of runs:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
10
Consistent fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
10 btrfs/186
10 btrfs/193
10 btrfs/200
10 btrfs/202
10 btrfs/204
10 btrfs/205
10 btrfs/214
10 btrfs/219
10 btrfs/220
10 btrfs/221
10 btrfs/222
10 generic/260
10 generic/286
10 generic/301
10 generic/371
10 generic/405
10 generic/407
10 generic/465
10 generic/475 <<-- New but not regression; failed 9/10 in original kernel.
10 generic/480
10 generic/488
10 generic/502
10 generic/513
10 generic/526
10 generic/527
10 generic/546
10 generic/562
10 generic/565
Flaky fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
1 btrfs/028
1 generic/089
1 generic/102
1 generic/269
2 generic/166
2 generic/464
2 generic/476
4 btrfs/010
4 generic/074
4 generic/142
4 generic/143
4 generic/531
6 generic/133
8 generic/175 << Increased from 2/10 to 8/10 b...
| Mauricio Faria de Oliveira (mfo) wrote : | #8 |
Focal/xfstests
=====
"pre-original"
$ grep -h ^PLATFORM xfstests.log.* | sort -u
PLATFORM -- Linux/s390x mfo-s390x-focal 5.4.0-52-generic #57+test20201026b1 SMP Mon Oct 26 19:12:47 -03 2020
PLATFORM -- Linux/s390x mfo-s390x-focal 5.4.0-52-generic #57+test20201028b1 SMP Wed Oct 28 11:21:07 -03 2020
PLATFORM -- Linux/s390x mfo-s390x-focal 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:52:40 UTC 2020
In Focal there are 8 regressions between master/
However, there are 0 regressions between master-next and master-next + this patchset. So, this patchset is good.
"pre-Original" kernel (master/
---
$ VERSION=
Number of runs:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
9
$ RUNS=9
Consistent fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
9 btrfs/153
9 btrfs/200
9 btrfs/204
9 btrfs/205
9 btrfs/213
9 btrfs/220
9 btrfs/221
9 btrfs/222
9 btrfs/226
9 generic/260
9 generic/301
9 generic/484
9 generic/610
Flaky fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
1 btrfs/010
1 generic/074
1 generic/127
1 generic/142
1 generic/143
1 generic/198
1 generic/207
1 generic/208
1 generic/209
1 generic/214
1 generic/215
1 generic/240
1 generic/241
1 generic/246
1 generic/371
2 generic/286
2 generic/465
3 generic/166
3 generic/564
4 generic/133
5 generic/175
7 generic/247
7 generic/475
Original kernel (master-next)
---
$ VERSION=
Number of runs:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
13
$ RUNS=13
Consistent fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
13 btrfs/153
13 btrfs/200
13 btrfs/204
13 btrfs/205
13 btrfs/213
13 btrfs/220
13 btrfs/221
13 btrfs/222
13 btrfs/226
13 generic/260
13 generic/286 << Regression; failed 2/9 in pre-original kernel.
13 generic/301
13 generic/465 << Regression; failed 2/9 in pre-original kernel.
13 generic/484
13 generic/610
Flaky fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
1 btrfs/001
1 btrfs/057
1 btrfs/058
1 btrfs/059
1 btrfs/060
...
| Mauricio Faria de Oliveira (mfo) wrote : | #9 |
Groovy/xfstests
======
Original/patched kernel versions:
$ grep -h ^PLATFORM xfstests.log.* | sort -u
PLATFORM -- Linux/s390x mfo-s390x-groovy 5.8.0-25-generic #26+test20201026b1 SMP Mon Oct 26 19:02:54 -03 2020
PLATFORM -- Linux/s390x mfo-s390x-groovy 5.8.0-25-generic #26-Ubuntu SMP Thu Oct 15 10:28:46 UTC 2020
Original kernel:
---
$ VERSION=
Number of runs:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
12
$ RUNS=12
Consistent fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
12 btrfs/153
12 btrfs/208
12 btrfs/213
12 btrfs/219
12 btrfs/220
12 btrfs/221
12 btrfs/222
12 generic/260
12 generic/286 <<-- Fails 15/18 in patched kernel
12 generic/301
12 generic/465 <<-- Fails 15/18 in patched kernel
12 generic/610
Flaky fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
1 generic/297
1 generic/371
1 generic/528
2 generic/298
3 generic/166
10 btrfs/010
10 generic/074
10 generic/142
10 generic/143
11 generic/133
11 generic/175
11 generic/475
Patched kernel:
---
$ VERSION=
Number of runs:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
18
$ RUNS=18
Consistent fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
18 btrfs/153
18 btrfs/208
18 btrfs/213
18 btrfs/219
18 btrfs/220
18 btrfs/221
18 btrfs/222
18 generic/260
18 generic/301
18 generic/610
No regressions.
Flaky fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
1 btrfs/020
1 btrfs/028
1 generic/118
1 generic/198
1 generic/215
1 generic/241
1 generic/246
1 generic/297
1 generic/371
1 generic/564
2 generic/298
3 generic/247
5 generic/166
11 btrfs/010
11 generic/074
11 generic/142
11 generic/143
13 generic/475
14 generic/175
15 generic/286
15 generic/465
16 generic/133
No regressions.
| Mauricio Faria de Oliveira (mfo) wrote : | #10 |
[B/F/G][PATCH 0/7] btrfs: Fix kernel BUG at fs/btrfs/
https:/
| tags: | added: architecture-s39064 bugnameltc-188988 severity-high targetmilestone-inin2004 |
| Changed in linux (Ubuntu Groovy): | |
| status: | In Progress → Fix Committed |
| Changed in linux (Ubuntu Focal): | |
| status: | In Progress → Fix Committed |
| Changed in linux (Ubuntu Bionic): | |
| status: | In Progress → Fix Committed |
| Kamal Mostafa (kamalmostafa) wrote : | #11 |
This mainline commit, recently backported to [B,F,G] for this bug:
"btrfs: extent-tree: kill BUG_ON() in __btrfs_
introduces a compile warning.
The warning has been fixed in mainline by:
cad69d139651 btrfs: tree-checker: fix incorrect printk format
so let's pick up that fix too.
Patch submitted: https:/
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
| tags: | added: verification-needed-bionic |
| tags: | added: verification-needed-focal |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
| tags: | added: verification-needed-groovy |
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-
If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.
See https:/
| Mauricio Faria de Oliveira (mfo) wrote : | #15 |
Verification done for groovy-proposed.
No regressions in consistent/flaky fail set.
Proposed kernel:
---
$ uname -rv
5.8.0-30-generic #32-Ubuntu SMP Mon Nov 9 21:02:16 UTC 2020
$ VERSION=
Number of runs:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
12
$ RUNS=12
Consistent fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
12 btrfs/153
12 btrfs/208
12 btrfs/213
12 btrfs/219
12 btrfs/220
12 btrfs/221
12 btrfs/222
12 generic/133 << not a regression; failed 11/12 in original kernel (comment #9)
12 generic/260
12 generic/286
12 generic/301
12 generic/465
12 generic/610
Flaky fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
1 btrfs/020
1 generic/298
9 generic/175
10 generic/475
11 btrfs/010
11 generic/074
11 generic/142
11 generic/143
| tags: |
added: verification-done-groovy removed: verification-needed-groovy |
| Mauricio Faria de Oliveira (mfo) wrote : | #16 |
Verification done for focal-proposed.
No regressions in the consistent/flaky fail sets.
Proposed kernel:
---
$ uname -rv
5.4.0-55-generic #61-Ubuntu SMP Mon Nov 9 20:45:34 UTC 2020
$ VERSION=
Number of runs:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
11
$ RUNS=11
Consistent fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
11 btrfs/153
11 btrfs/205
11 btrfs/213
11 btrfs/220
11 btrfs/221
11 btrfs/222
11 generic/133 << not a regression; failed 12/13 in original master-next
11 generic/260
11 generic/286
11 generic/301
11 generic/465
11 generic/484
11 generic/610
btrfs/200 fixed.
btrfs/204 fixed.
btrfs/226 fixed.
Flaky fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
1 generic/297
1 generic/371
3 generic/166
9 generic/175
10 btrfs/010
10 generic/074
10 generic/142
10 generic/143
10 generic/475
| tags: |
added: verification-done-focal removed: verification-needed-focal |
| Mauricio Faria de Oliveira (mfo) wrote : | #17 |
Verification done for focal-proposed.
No regressions in the consistent/flaky fail sets.
The flaky fail set had a long tail (tests that failed just a few times),
but it can be isolated to another btrfs commit, and it doesn't seem to
happen often. (e.g. not seen on focal-proposed, but seen on focal orig
kernel on master-next, and both have the commit.)
Thus not a general/permanent issue. More details in the next comment.
Proposed kernel:
---
$ uname -rv
4.15.0-125-generic #128-Ubuntu SMP Mon Nov 9 20:45:41 UTC 2020
$ VERSION=
Number of runs:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
14
$ RUNS=14
Consistent fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
14 btrfs/186
14 btrfs/193
14 btrfs/200
14 btrfs/202
14 btrfs/204
14 btrfs/205
14 btrfs/214
14 btrfs/219
14 btrfs/220
14 btrfs/221
14 btrfs/222
14 generic/260
14 generic/286
14 generic/301
14 generic/371
14 generic/405
14 generic/407
14 generic/465
14 generic/480
14 generic/488
14 generic/502
14 generic/513
14 generic/526
14 generic/527
14 generic/546
14 generic/562
14 generic/565
Flaky fail set:
$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
1 btrfs/001
1 btrfs/147
1 btrfs/148
1 btrfs/149
1 btrfs/150
1 btrfs/151
1 btrfs/152
1 btrfs/153
1 btrfs/155
1 btrfs/156
1 btrfs/159
1 btrfs/164
1 btrfs/165
1 btrfs/166
1 btrfs/168
1 btrfs/169
1 btrfs/170
1 btrfs/171
1 btrfs/172
1 btrfs/173
1 btrfs/174
1 btrfs/177
1 btrfs/178
1 btrfs/179
1 btrfs/180
1 btrfs/181
1 btrfs/182
1 btrfs/183
1 btrfs/184
1 btrfs/187
1 btrfs/188
1 btrfs/190
1 btrfs/196
1 btrfs/197
1 btrfs/199
1 btrfs/206
1 btrfs/207
1 btrfs/208
1 btrfs/213
1 btrfs/217
1 generic/003
1 generic/017
1 generic/031
1 generic/033
1 generic/037
1 generic/038
1 generic/039
1 generic/040
1 generic/041
1 generic/043
1 generic/044
1 generic/045
1 generic/046
1 generic/047
1 generic/048
1 generic/049
1 generic/050
1 generic/051
1 generic/052
1 generic/053
1 generic/054
1 generic/055
1 generic/057
1 generic/059
1 generic/062
1 generic/064
1 generic/065
1 generic/066
1 generic/067
1 generic/068
1 generic/069
1 generic/071
1 generic/073
1 generic/076
1 generic/077
1 generic/079
1 generic/081
1 generic/082
1 generic/083
1 generic/084
1 generic/090
1 generic/1...
| tags: |
added: verification-done-bionic removed: verification-needed-bionic |
| Mauricio Faria de Oliveira (mfo) wrote : | #18 |
In comment #17, read:
Verification done for BIONIC-proposed. (see 4.15 kernel.)
| Mauricio Faria de Oliveira (mfo) wrote : | #19 |
Explanatory note on bionic-proposed (see comment #17):
The flaky fail set had a long tail (tests that failed just a few times).
Symptom:
---
The logs show the failure cause is output mismatch, because '/target'
failed to umount (busy), and it was logged; e.g.,
btrfs/027 [failed, exit status 1]- output mismatch (see /home/ubuntu/
--- tests/btrfs/027.out 2020-10-26 22:23:54.626341276 +0000
+++ /home/ubuntu/
@@ -1,2 +1,3 @@
QA output created by 027
-Silence is golden
+umount: /mnt/scratch: target is busy.
+failed to unmount /dev/loop2
...
The number of failed tests increases with the number of umount failures.
Comparing both numbers per log file: (line breaks added for clarity.)
$ (grep ^Failed xfstests.
grep -c 'umount: /mnt/scratch: target is busy' xfstests.
| sort
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
xfstests.
And the kernel log files show that 'umount' hung in the timestamps where the increase started.
$ sudo grep umount kern.log.
kern.log.
kern.log.
kern.log.
kern.log.
kern.log.
| Launchpad Janitor (janitor) wrote : | #20 |
This bug was fixed in the package linux - 5.4.0-56.62
---------------
linux (5.4.0-56.62) focal; urgency=medium
* focal/linux: 5.4.0-56.62 -proposed tracker (LP: #1905300)
* CVE-2020-4788
- selftests/powerpc: rfi_flush: disable entry flush if present
- powerpc/64s: flush L1D on kernel entry
- powerpc/64s: flush L1D after user accesses
- selftests/powerpc: entry flush test
linux (5.4.0-55.61) focal; urgency=medium
* focal/linux: 5.4.0-55.61 -proposed tracker (LP: #1903175)
* Update kernel packaging to support forward porting kernels (LP: #1902957)
- [Debian] Update for leader included in BACKPORT_SUFFIX
* Avoid double newline when running insertchanges (LP: #1903293)
- [Packaging] insertchanges: avoid double newline
* EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
- efivarfs: Replace invalid slashes with exclamation marks in dentries.
* CVE-2020-14351
- perf/core: Fix race in the perf_mmap_close() function
* raid10: Block discard is very slow, causing severe delays for mkfs and
fstrim operations (LP: #1896578)
- md: add md_submit_
- md/raid10: extend r10bio devs to raid disks
- md/raid10: pull codes that wait for blocked dev into one function
- md/raid10: improve raid10 discard request
- md/raid10: improve discard request for far layout
- dm raid: fix discard limits for raid1 and raid10
- dm raid: remove unnecessary discard limits for raid10
* Bionic: btrfs: kernel BUG at /build/linux-
eTBZpZ/
- btrfs: drop unnecessary offset_in_page in extent buffer helpers
- btrfs: extent_io: do extra check for extent buffer read write functions
- btrfs: extent-tree: kill BUG_ON() in __btrfs_
- btrfs: extent-tree: kill the BUG_ON() in insert_
- btrfs: ctree: check key order before merging tree blocks
* Ethernet no link lights after reboot (Intel i225-v 2.5G) (LP: #1902578)
- igc: Add PHY power management control
* Undetected Data corruption in MPI workloads that use VSX for reductions on
POWER9 DD2.1 systems (LP: #1902694)
- powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
- selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
workaround
* [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
- s390: nvme ipl
- s390: nvme reipl
- s390/ipl: support NVMe IPL kernel parameters
* uvcvideo: add mapping for HEVC payloads (LP: #1895803)
- media: uvcvideo: Add mapping for HEVC payloads
* Focal update: v5.4.73 upstream stable release (LP: #1902115)
- ibmveth: Switch order of ibmveth_helper calls.
- ibmveth: Identify ingress large send packets.
- ipv4: Restore flowi4_oif update before call to xfrm_lookup_route
- mlx4: handle non-napi callers to napi_poll
- net: fec: Fix phy_device lookup for phy_reset_
- net: fec: Fix PHY init after phy_reset_
- net: fix pos incrementment in ipv6_route_seq_next
- net/smc: fix valid DMBE buffer sizes
- net...
| Changed in linux (Ubuntu Focal): | |
| status: | Fix Committed → Fix Released |
| Launchpad Janitor (janitor) wrote : | #21 |
This bug was fixed in the package linux - 5.8.0-31.33
---------------
linux (5.8.0-31.33) groovy; urgency=medium
* groovy/linux: 5.8.0-31.33 -proposed tracker (LP: #1905299)
* Groovy 5.8 kernel hangs on boot on CPUs with eLLC (LP: #1903397)
- drm/i915: Mark ininitial fb obj as WT on eLLC machines to avoid rcu lockup
during fbdev init
* CVE-2020-4788
- selftests/powerpc: rfi_flush: disable entry flush if present
- powerpc/64s: flush L1D on kernel entry
- powerpc/64s: flush L1D after user accesses
- selftests/powerpc: entry flush test
linux (5.8.0-30.32) groovy; urgency=medium
* groovy/linux: 5.8.0-30.32 -proposed tracker (LP: #1903194)
* Update kernel packaging to support forward porting kernels (LP: #1902957)
- [Debian] Update for leader included in BACKPORT_SUFFIX
* Avoid double newline when running insertchanges (LP: #1903293)
- [Packaging] insertchanges: avoid double newline
* EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
- efivarfs: Replace invalid slashes with exclamation marks in dentries.
* raid10: Block discard is very slow, causing severe delays for mkfs and
fstrim operations (LP: #1896578)
- md: add md_submit_
- md/raid10: extend r10bio devs to raid disks
- md/raid10: pull codes that wait for blocked dev into one function
- md/raid10: improve raid10 discard request
- md/raid10: improve discard request for far layout
- dm raid: fix discard limits for raid1 and raid10
- dm raid: remove unnecessary discard limits for raid10
* Bionic: btrfs: kernel BUG at /build/linux-
eTBZpZ/
- btrfs: extent_io: do extra check for extent buffer read write functions
- btrfs: extent-tree: kill BUG_ON() in __btrfs_
- btrfs: extent-tree: kill the BUG_ON() in insert_
- btrfs: ctree: check key order before merging tree blocks
* Tiger Lake PMC core driver fixes (LP: #1899883)
- platform/x86: intel_pmc_core: update TGL's LPM0 reg bit map name
- platform/x86: intel_pmc_core: fix bound check in pmc_core_
- platform/x86: pmc_core: Use descriptive names for LPM registers
- platform/x86: intel_pmc_core: Fix TigerLake power gating status map
- platform/x86: intel_pmc_core: Fix the slp_s0 counter displayed value
* drm/i915/dp_mst - System would hang during the boot up. (LP: #1902469)
- Revert "UBUNTU: SAUCE: drm/i915/display: Fix null deref in
intel_
- drm/i915: Fix encoder lookup during PSR atomic check
* Undetected Data corruption in MPI workloads that use VSX for reductions on
POWER9 DD2.1 systems (LP: #1902694)
- powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
- selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
workaround
* [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
- s390/ipl: support NVMe IPL kernel parameters
* uvcvideo: add mapping for HEVC payloads (LP: #1895803)
- media: uvcvideo: Add mapping for HEVC payloads
* risc-v 5.8 ...
| Changed in linux (Ubuntu Groovy): | |
| status: | Fix Committed → Fix Released |
| Launchpad Janitor (janitor) wrote : | #22 |
This bug was fixed in the package linux - 4.15.0-126.129
---------------
linux (4.15.0-126.129) bionic; urgency=medium
* bionic/linux: 4.15.0-126.129 -proposed tracker (LP: #1905305)
* CVE-2020-4788
- SAUCE: powerpc/64s: Define MASKABLE_
- SAUCE: powerpc/64s: move some exception handlers out of line
- powerpc/64s: flush L1D on kernel entry
- SAUCE: powerpc: Add a framework for user access tracking
- powerpc: Implement user_access_begin and friends
- powerpc: Fix __clear_user() with KUAP enabled
- powerpc/uaccess: Evaluate macro arguments once, before user access is
allowed
- powerpc/64s: flush L1D after user accesses
linux (4.15.0-125.128) bionic; urgency=medium
* bionic/linux: 4.15.0-125.128 -proposed tracker (LP: #1903137)
* Update kernel packaging to support forward porting kernels (LP: #1902957)
- [Debian] Update for leader included in BACKPORT_SUFFIX
* Avoid double newline when running insertchanges (LP: #1903293)
- [Packaging] insertchanges: avoid double newline
* EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
- efivarfs: Replace invalid slashes with exclamation marks in dentries.
* CVE-2020-14351
- perf/core: Fix race in the perf_mmap_close() function
* raid10: Block discard is very slow, causing severe delays for mkfs and
fstrim operations (LP: #1896578)
- md: add md_submit_
- md/raid10: extend r10bio devs to raid disks
- md/raid10: pull codes that wait for blocked dev into one function
- md/raid10: improve raid10 discard request
- md/raid10: improve discard request for far layout
* Bionic: btrfs: kernel BUG at /build/linux-
eTBZpZ/
- btrfs: use offset_in_page instead of open-coding it
- btrfs: use BUG() instead of BUG_ON(1)
- btrfs: drop unnecessary offset_in_page in extent buffer helpers
- btrfs: extent_io: do extra check for extent buffer read write functions
- btrfs: extent-tree: kill BUG_ON() in __btrfs_
- btrfs: extent-tree: kill the BUG_ON() in insert_
- btrfs: ctree: check key order before merging tree blocks
* Bionic update: upstream stable patchset 2020-11-04 (LP: #1902943)
- USB: gadget: f_ncm: Fix NDP16 datagram validation
- gpio: tc35894: fix up tc35894 interrupt configuration
- vsock/virtio: use RCU to avoid use-after-free on the_virtio_vsock
- vsock/virtio: stop workers during the .remove()
- vsock/virtio: add transport parameter to the
virtio_
- net: virtio_vsock: Enhance connection semantics
- Input: i8042 - add nopnp quirk for Acer Aspire 5 A515
- ftrace: Move RCU is watching check after recursion check
- drm/amdgpu: restore proper ref count in amdgpu_
- drivers/
- drm/sun4i: mixer: Extend regmap max_register
- net: dec: de2104x: Increase receive ring size for Tulip
- rndis_host: increase sleep time in the query-response loop
- nvme-core: get/put ctrl ...
| Changed in linux (Ubuntu Bionic): | |
| status: | Fix Committed → Fix Released |
| Changed in ubuntu-z-systems: | |
| status: | New → Fix Released |
------- Comment From <email address hidden> 2020-12-02 03:36 EDT-------
IBM Bugzilla status-> closed, Fix Released by Canonical
| Launchpad Janitor (janitor) wrote : | #24 |
This bug was fixed in the package linux - 5.8.0-36.40+21.04.1
---------------
linux (5.8.0-
* Packaging resync (LP: #1786013)
- update dkms package versions
[ Ubuntu: 5.8.0-36.40 ]
* debian/
(LP: #1878897)
- [Packaging] file-downloader not handling positive failures correctly
[ Ubuntu: 5.8.0-35.39 ]
* Packaging resync (LP: #1786013)
- update dkms package versions
* CVE-2021-1052 // CVE-2021-1053
- [Packaging] NVIDIA -- Add the NVIDIA 460 driver
-- Kleber Sacilotto de Souza <email address hidden> Thu, 07 Jan 2021 11:57:30 +0100
| Changed in linux (Ubuntu): | |
| status: | Fix Committed → Fix Released |
[224259.453356] ------------[ cut here ]------------
[224259.453360] kernel BUG at /build/
[224259.453390] illegal operation: 0001 ilc:1 [#1] SMP
[224259.453392] Modules linked in: vhost_net xt_nat macvtap tap veth macvlan ipt_MASQUERADE nf_nat_
type iptable_nat nf_nat_ipv4 nf_nat br_netfilter bridge vhost_vsock vmw_vsock_
021q garp stp mrp llc overlay nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip6
table_filter ip6_tables qeth_l2 s390_trng qeth chsc_sch ccwgroup vfio_ccw eadm_sch iptable_filter sch_fq_codel zFPC_proc(OE) zFPC_diag(OE) vfio_ap vfio_mdev mdev vfio_iommu_
type1 vfio ip_tables x_tables ghash_s390 prng aes_s390 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common
[224259.453415] crc32_vx_s390 btrfs xor zstd_compress raid6_pq dm_crypt virtio_blk dm_service_time dm_multipath zfcp scsi_transport_fc qdio dasd_eckd_mod dasd_mod zlib_defl
ate
[224259.453423] CPU: 6 PID: 57332 Comm: qemu-system-s39 Tainted: G OE 4.15.0-109-generic #110-Ubuntu
[224259.453423] Hardware name: IBM 3907 LR1 A00 (LPAR)
[224259.453425] Krnl PSW : 00000000be9cb874 00000000ef3786e8 (btrfs_
[224259.453492] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[224259.453493] Krnl GPRS: 0000000000000001 00000056bdc00000 0000000000000000 0000000cbf93795d
[224259.453493] 000003ff0000006c 0000000000631468 000000136a230000 0000000cbf937826
[224259.453494] 00000007ee281d88 0000000cbf93795d 00000007000000a1 00000013188a58c0
[224259.453495] 00000012eb889701 00000056bdb00000 000003ff801780a8 0000000cbf937780
[224259.453503] Krnl Code: 000003ff801780a2: c0e5fffff857 brasl %r14,000003ff80
000003ff801780a8: ec22ff92007e cij %r2,0,2,
#000003ff801780ae: a7f40001 brc 15,000003ff801780b0
>000003ff801780b2: ec2affff00d8 ahik %r2,%r10,-1
000003ff801780b8: b9140022 lgfr %r2,%r2
000003ff801780bc: eb120001000d sllg %r1,%r2,1
000003ff801780c2: b9080012 agr %r1,%r2
000003ff801780c6: eb110003000d sllg %r1,%r1,3
[224259.453514] Call Trace:
[224259.453527] ([<000003ff8017
[224259.453544] [<000003ff801c3
[224259.453561] [<000003ff801f9
[224259.453577] [<000003ff801fa
[224259.453594] [<000003ff801fb
[224259.453611] [<000003ff801fc
[224259.453627] [<000003ff801c5
[224259.453634] [<00000000003cc
[224259.453636] [<00000000003cd
[224259.453640] [<00000000008fd
[224259.453640] Last Breaking-
[224259.453652] [<000003ff80178
[224259.453653]
[224259.4536...