Bionic: btrfs: kernel BUG at /build/linux-eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233!

[Mauricio Faria de Oliveira]

[Impact]

 * Users of btrfs started hitting a kernel BUG() (below)
   after upgrade from 4.15.0-99.100 to 4.15.0-109.110,
   which has 55 btrfs changes.

     kernel BUG at /build/linux-eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233!
     ...
     Krnl PSW : 00000000be9cb874 00000000ef3786e8 (btrfs_set_item_key_safe+0x152/0x1c0 [btrfs])
     ...
     [...] Call Trace:
     [...] btrfs_set_item_key_safe+0x11c/0x1c0 [btrfs])
     [...] __btrfs_drop_extents+0xb5a/0xda8 [btrfs]
     [...] btrfs_log_changed_extents+0x35c/0xaf0 [btrfs]
     [...] btrfs_log_inode+0x9ee/0x1080 [btrfs]
     [...] btrfs_log_inode_parent+0x224/0xa10 [btrfs]
     [...] btrfs_log_dentry_safe+0x80/0xa8 [btrfs]
     [...] btrfs_sync_file+0x392/0x550 [btrfs]
     [...] do_fsync+0x5e/0x90
     [...] SyS_fdatasync+0x32/0x48
     [...] system_call+0xd8/0x2c8

     $ git log --oneline Ubuntu-4.15.0-99.100..Ubuntu-4.15.0-109.110 -- fs/btrfs/ | wc -l
     55

 * The error happens at random moments, regardless of a
   particular activity/load. Workaround is to downgrade.

[Fix]

 * This BUG()/function is addressed in patch 4/4 [1] of series
   'btrfs: Enhanced runtime defence against fuzzed images' [2],
   after issues in the real world, not just crafted fs images:
   'one internal report has hit one BUG_ON() with real world fs'

     kernel BUG at fs/btrfs/ctree.c:3188!
     ...
     RIP: 0010:btrfs_set_item_key_safe+0x16c/0x180

 * The patch/set [3] is applied in v5.10-rc1 and Ubuntu Unstable:
   - d16c702fe4f2 btrfs: ctree: check key order before merging tree blocks
   - 07cce5cf3b48 btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
   - 1c2a07f598d5 btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
   - f98b6215d7d1 btrfs: extent_io: do extra check for extent buffer read write functions

[Test Case]

 * There is working synthetic reproducer for this issue,
   which is hard to reproduce as reported in commit [4]
   that introduces debugging for the issue.

 * Regression tests with xfstests and stress-ng shows
   no regressions between un/patched kernels.

[Other Info]

 * Trivial backports (only refreshing a few context lines)
   with 3 more dependency patches on Bionic and 1 on Focal.
   And Bionic needed one extra hunk to '#include' a header.
   Groovy all apply cleanly.

[1] https://<email address hidden>/
[2] https://<email address hidden>/
[3] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=d16c702fe4f274bd77b47d3ab737eadcf24e0b93
[4] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=7c15d41016dc886cc011e3854d855e219759ae68

[Mauricio Faria de Oliveira (mfo)]

[224259.453356] ------------[ cut here ]------------
[224259.453360] kernel BUG at /build/linux-eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233!
[224259.453390] illegal operation: 0001 ilc:1 [#1] SMP
[224259.453392] Modules linked in: vhost_net xt_nat macvtap tap veth macvlan ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addr
type iptable_nat nf_nat_ipv4 nf_nat br_netfilter bridge vhost_vsock vmw_vsock_virtio_transport_common vhost vsock algif_skcipher af_alg xt_tcpudp xt_multiport aufs bonding 8
021q garp stp mrp llc overlay nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c scsi_dh_rdac scsi_dh_emc scsi_dh_alua ip6
table_filter ip6_tables qeth_l2 s390_trng qeth chsc_sch ccwgroup vfio_ccw eadm_sch iptable_filter sch_fq_codel zFPC_proc(OE) zFPC_diag(OE) vfio_ap vfio_mdev mdev vfio_iommu_
type1 vfio ip_tables x_tables ghash_s390 prng aes_s390 des_s390 des_generic sha512_s390 sha256_s390 sha1_s390 sha_common
[224259.453415] crc32_vx_s390 btrfs xor zstd_compress raid6_pq dm_crypt virtio_blk dm_service_time dm_multipath zfcp scsi_transport_fc qdio dasd_eckd_mod dasd_mod zlib_defl
ate
[224259.453423] CPU: 6 PID: 57332 Comm: qemu-system-s39 Tainted: G OE 4.15.0-109-generic #110-Ubuntu
[224259.453423] Hardware name: IBM 3907 LR1 A00 (LPAR)
[224259.453425] Krnl PSW : 00000000be9cb874 00000000ef3786e8 (btrfs_set_item_key_safe+0x152/0x1c0 [btrfs])
[224259.453492] R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0 PM:0 RI:0 EA:3
[224259.453493] Krnl GPRS: 0000000000000001 00000056bdc00000 0000000000000000 0000000cbf93795d
[224259.453493] 000003ff0000006c 0000000000631468 000000136a230000 0000000cbf937826
[224259.453494] 00000007ee281d88 0000000cbf93795d 00000007000000a1 00000013188a58c0
[224259.453495] 00000012eb889701 00000056bdb00000 000003ff801780a8 0000000cbf937780
[224259.453503] Krnl Code: 000003ff801780a2: c0e5fffff857 brasl %r14,000003ff80177150
000003ff801780a8: ec22ff92007e cij %r2,0,2,000003ff80177fcc
#000003ff801780ae: a7f40001 brc 15,000003ff801780b0
>000003ff801780b2: ec2affff00d8 ahik %r2,%r10,-1
000003ff801780b8: b9140022 lgfr %r2,%r2
000003ff801780bc: eb120001000d sllg %r1,%r2,1
000003ff801780c2: b9080012 agr %r1,%r2
000003ff801780c6: eb110003000d sllg %r1,%r1,3
[224259.453514] Call Trace:
[224259.453527] ([<000003ff8017807c>] btrfs_set_item_key_safe+0x11c/0x1c0 [btrfs])
[224259.453544] [<000003ff801c3322>] __btrfs_drop_extents+0xb5a/0xda8 [btrfs]
[224259.453561] [<000003ff801f98a4>] btrfs_log_changed_extents+0x35c/0xaf0 [btrfs]
[224259.453577] [<000003ff801faa26>] btrfs_log_inode+0x9ee/0x1080 [btrfs]
[224259.453594] [<000003ff801fb384>] btrfs_log_inode_parent+0x224/0xa10 [btrfs]
[224259.453611] [<000003ff801fcea8>] btrfs_log_dentry_safe+0x80/0xa8 [btrfs]
[224259.453627] [<000003ff801c5ea2>] btrfs_sync_file+0x392/0x550 [btrfs]
[224259.453634] [<00000000003cce5e>] do_fsync+0x5e/0x90
[224259.453636] [<00000000003cd15a>] SyS_fdatasync+0x32/0x48
[224259.453640] [<00000000008fd314>] system_call+0xd8/0x2c8
[224259.453640] Last Breaking-Event-Address:
[224259.453652] [<000003ff801780ae>] btrfs_set_item_key_safe+0x14e/0x1c0 [btrfs]
[224259.453653]
[224259.4536...

[Mauricio Faria de Oliveira (mfo)]

Regression testing done with 3 approaches:

1) synthetic reproducer in kernel.org BZ#202833 [1]
2) stress-ng --class filesystem,io
3) xfstests (aka fstests)

No regressions observed in Bionic/Focal/Groovy between original/patched kernels, in s390x native VMs in canonistack-bos01.

[Mauricio Faria de Oliveira (mfo)]

1) synthetic reproducer

The reproducer consists in a crafted disk/btrfs image and C test-case.

This has not been helpful.
In Bionic, the crafted disk/btrfs image does not even mount.
In Focal/Groovy it does mount, but the test-case does not trigger any errors.

Anyway, the behavior is the same in the original/patched kernels.
No regressions.

[Mauricio Faria de Oliveira (mfo)]

2) stress-ng --class filesystem,io

Compared the stress-ng output and kernel log between original/patched kernels.
No regressions.

Actually, in Bionic the original kernel didn't even finish stress-ng in ~20 hours, while the patched kernel finished it in ~1.5 hours (in line w/ other the releases.)
Thus it's apparently an improvement.

For documentation purposes, this seems to be due to the 'fiemap' stressor in the 'filesystem' class, which had defunct processes when I checked before cancelling the tests:

 $ ps axf
 ...
  2659 ? Ss 0:00 tmux new
  2661 pts/2 Ss 0:00 \_ -bash
  2692 pts/2 S+ 0:00 | \_ /bin/sh ./run-stressng.sh
  2743 pts/2 S+ 0:00 | \_ sudo stress-ng --sequential 2 --class filesystem,io -t 2m
  2745 pts/2 SL+ 0:00 | \_ stress-ng --sequential 2 --class filesystem io -t 2m
  2852 pts/2 D+ 0:00 | \_ stress-ng --sequential 2 --class filesystem io -t 2m
  2854 pts/2 D+ 0:01 | | \_ stress-ng --sequential 2 --class filesystem io -t 2m
  2856 pts/2 D+ 0:01 | | \_ stress-ng --sequential 2 --class filesystem io -t 2m
  2859 pts/2 D+ 0:01 | | \_ stress-ng --sequential 2 --class filesystem io -t 2m
  2860 pts/2 D+ 0:01 | | \_ stress-ng --sequential 2 --class filesystem io -t 2m
  2853 pts/2 D+ 0:00 | \_ stress-ng --sequential 2 --class filesystem io -t 2m
  2855 pts/2 Z+ 322:29 | \_ [stress-ng-fiema] <defunct>
  2857 pts/2 Z+ 323:55 | \_ [stress-ng-fiema] <defunct>
  2858 pts/2 Z+ 321:50 | \_ [stress-ng-fiema] <defunct>
  2861 pts/2 Z+ 322:51 | \_ [stress-ng-fiema] <defunct>
 11823 pts/4 Ss 0:00 \_ -bash
 ...

[Mauricio Faria de Oliveira (mfo)]

[1] kernel.org BZ#202833
    https://bugzilla.kernel.org/show_bug.cgi?id=202833

[Mauricio Faria de Oliveira (mfo)]

3) xfstests (aka fstests)

Compared the 'consistently fail set' and 'likelyhood to fail' in flaky tests (see below.)
No regressions.

Details:
---

Source:
- git://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git
- commit 31f6949f ("ext4: verify unwritten extent conversion in buff-io")

The test set is the 'full' suite (default), not just the 'quick' group/suite.

The fail set is flaky (some tests fail/pass inconsistently), so the 'consistently fail set' (i.e., fail every time) across 10-20 runs has been compared between the original/patched kernels.

Also looked for increased likelyhood to fail (i.e., flaky tests that fail more often with the patches.)

More Details:
---

The xfstests output/log includes the fail set in the 'Failures:' line, and the kernel version in the 'PLATFORM' line.
For example:

 $ grep -e ^Fail -e ^PLATFORM xfstests.log.2020-10-27-13-26-44
 PLATFORM -- Linux/s390x mfo-s390x-focal 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:52:40 UTC 2020
 Failures: btrfs/153 btrfs/200 btrfs/204 btrfs/205 btrfs/213 btrfs/220 btrfs/221 btrfs/222 btrfs/226 generic/166 generic/175 generic/260 generic/286 generic/301 generic/465 generic/484 generic/610

Those two lines from each log/run are used to identify, per kernel version (original/patched):
- how many runs have been done;
- which tests fail consistently (i.e., in all runs);
- which tests fail inconsistently (i.e., flaky) and how many times (to compare likelyhood to fail)

[Mauricio Faria de Oliveira (mfo)]

Bionic/xfstests
======

Original/patched kernel versions:

 $ grep -h ^PLATFORM xfstests.log.* | sort -u
 PLATFORM -- Linux/s390x mfo-s390x-bionic 4.15.0-122-generic #124+test20201026b1 SMP Mon Oct 26 13:10:37 -03 2020
 PLATFORM -- Linux/s390x mfo-s390x-bionic 4.15.0-122-generic #124-Ubuntu SMP Thu Oct 15 13:02:05 UTC 2020

Original kernel:
---

 $ VERSION='4.15.0-122-generic #124-Ubuntu'

Number of runs:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
 10

 $ RUNS=10

Consistent fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
      10 btrfs/186
      10 btrfs/193
      10 btrfs/200
      10 btrfs/202
      10 btrfs/204
      10 btrfs/205
      10 btrfs/214
      10 btrfs/219
      10 btrfs/220
      10 btrfs/221
      10 btrfs/222
      10 generic/260
      10 generic/286
      10 generic/301
      10 generic/371
      10 generic/405
      10 generic/407
      10 generic/465
      10 generic/480
      10 generic/488
      10 generic/502
      10 generic/513
      10 generic/526
      10 generic/527
      10 generic/546
      10 generic/562
      10 generic/565

Flaky fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
       1 btrfs/028
       1 generic/089
       1 generic/464
       2 generic/175
       3 generic/476
       4 generic/531
       7 btrfs/078
       8 btrfs/010
       8 generic/074
       8 generic/142
       8 generic/143
       9 generic/133
       9 generic/475

Patched kernel:
---

 $ VERSION='4.15.0-122-generic #124+test20201026b1'

Number of runs:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
 10

Consistent fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
      10 btrfs/186
      10 btrfs/193
      10 btrfs/200
      10 btrfs/202
      10 btrfs/204
      10 btrfs/205
      10 btrfs/214
      10 btrfs/219
      10 btrfs/220
      10 btrfs/221
      10 btrfs/222
      10 generic/260
      10 generic/286
      10 generic/301
      10 generic/371
      10 generic/405
      10 generic/407
      10 generic/465
      10 generic/475 <<-- New but not regression; failed 9/10 in original kernel.
      10 generic/480
      10 generic/488
      10 generic/502
      10 generic/513
      10 generic/526
      10 generic/527
      10 generic/546
      10 generic/562
      10 generic/565

Flaky fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
       1 btrfs/028
       1 generic/089
       1 generic/102
       1 generic/269
       2 generic/166
       2 generic/464
       2 generic/476
       4 btrfs/010
       4 generic/074
       4 generic/142
       4 generic/143
       4 generic/531
       6 generic/133
       8 generic/175 << Increased from 2/10 to 8/10 b...

[Mauricio Faria de Oliveira (mfo)]

Focal/xfstests
=====

"pre-original"/original/patched kernel versions:

 $ grep -h ^PLATFORM xfstests.log.* | sort -u
 PLATFORM -- Linux/s390x mfo-s390x-focal 5.4.0-52-generic #57+test20201026b1 SMP Mon Oct 26 19:12:47 -03 2020
 PLATFORM -- Linux/s390x mfo-s390x-focal 5.4.0-52-generic #57+test20201028b1 SMP Wed Oct 28 11:21:07 -03 2020
 PLATFORM -- Linux/s390x mfo-s390x-focal 5.4.0-52-generic #57-Ubuntu SMP Thu Oct 15 10:52:40 UTC 2020

In Focal there are 8 regressions between master/bionic-updates and master-next/not-yet-bionic-proposed (i.e., caused by other 13 btrfs patches; _not_ this patchset.)

However, there are 0 regressions between master-next and master-next + this patchset. So, this patchset is good.

"pre-Original" kernel (master/bionic-updates)
---

 $ VERSION='5.4.0-52-generic #57-Ubuntu'

Number of runs:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
 9

 $ RUNS=9

Consistent fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
       9 btrfs/153
       9 btrfs/200
       9 btrfs/204
       9 btrfs/205
       9 btrfs/213
       9 btrfs/220
       9 btrfs/221
       9 btrfs/222
       9 btrfs/226
       9 generic/260
       9 generic/301
       9 generic/484
       9 generic/610

Flaky fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
       1 btrfs/010
       1 generic/074
       1 generic/127
       1 generic/142
       1 generic/143
       1 generic/198
       1 generic/207
       1 generic/208
       1 generic/209
       1 generic/214
       1 generic/215
       1 generic/240
       1 generic/241
       1 generic/246
       1 generic/371
       2 generic/286
       2 generic/465
       3 generic/166
       3 generic/564
       4 generic/133
       5 generic/175
       7 generic/247
       7 generic/475

Original kernel (master-next)
---

 $ VERSION='5.4.0-52-generic #57+test20201028b1'

Number of runs:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
 13

 $ RUNS=13

Consistent fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
      13 btrfs/153
      13 btrfs/200
      13 btrfs/204
      13 btrfs/205
      13 btrfs/213
      13 btrfs/220
      13 btrfs/221
      13 btrfs/222
      13 btrfs/226
      13 generic/260
      13 generic/286 << Regression; failed 2/9 in pre-original kernel.
      13 generic/301
      13 generic/465 << Regression; failed 2/9 in pre-original kernel.
      13 generic/484
      13 generic/610

Flaky fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
       1 btrfs/001
       1 btrfs/057
       1 btrfs/058
       1 btrfs/059
       1 btrfs/060
   ...

[Mauricio Faria de Oliveira (mfo)]

Groovy/xfstests
======

Original/patched kernel versions:

 $ grep -h ^PLATFORM xfstests.log.* | sort -u
 PLATFORM -- Linux/s390x mfo-s390x-groovy 5.8.0-25-generic #26+test20201026b1 SMP Mon Oct 26 19:02:54 -03 2020
 PLATFORM -- Linux/s390x mfo-s390x-groovy 5.8.0-25-generic #26-Ubuntu SMP Thu Oct 15 10:28:46 UTC 2020

Original kernel:
---

 $ VERSION='5.8.0-25-generic #26-Ubuntu'

Number of runs:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
 12

 $ RUNS=12

Consistent fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
      12 btrfs/153
      12 btrfs/208
      12 btrfs/213
      12 btrfs/219
      12 btrfs/220
      12 btrfs/221
      12 btrfs/222
      12 generic/260
      12 generic/286 <<-- Fails 15/18 in patched kernel
      12 generic/301
      12 generic/465 <<-- Fails 15/18 in patched kernel
      12 generic/610

Flaky fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
       1 generic/297
       1 generic/371
       1 generic/528
       2 generic/298
       3 generic/166
      10 btrfs/010
      10 generic/074
      10 generic/142
      10 generic/143
      11 generic/133
      11 generic/175
      11 generic/475

Patched kernel:
---

 $ VERSION='5.8.0-25-generic #26+test20201026b1'

Number of runs:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
 18

 $ RUNS=18

Consistent fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
      18 btrfs/153
      18 btrfs/208
      18 btrfs/213
      18 btrfs/219
      18 btrfs/220
      18 btrfs/221
      18 btrfs/222
      18 generic/260
      18 generic/301
      18 generic/610

  No regressions.

Flaky fail set:

 $ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
       1 btrfs/020
       1 btrfs/028
       1 generic/118
       1 generic/198
       1 generic/215
       1 generic/241
       1 generic/246
       1 generic/297
       1 generic/371
       1 generic/564
       2 generic/298
       3 generic/247
       5 generic/166
      11 btrfs/010
      11 generic/074
      11 generic/142
      11 generic/143
      13 generic/475
      14 generic/175
      15 generic/286
      15 generic/465
      16 generic/133

  No regressions.

[Mauricio Faria de Oliveira (mfo)]

[B/F/G][PATCH 0/7] btrfs: Fix kernel BUG at fs/btrfs/ctree.c:3233 / btrfs_set_item_key_safe()
https://lists.ubuntu.com/archives/kernel-team/2020-October/114398.html

[Kamal Mostafa (kamalmostafa)]

This mainline commit, recently backported to [B,F,G] for this bug:
    "btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()"
introduces a compile warning.

The warning has been fixed in mainline by:
    cad69d139651 btrfs: tree-checker: fix incorrect printk format
so let's pick up that fix too.

Patch submitted: https://lists.ubuntu.com/archives/kernel-team/2020-November/114752.html

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed-groovy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Mauricio Faria de Oliveira (mfo)]

Verification done for groovy-proposed.
No regressions in consistent/flaky fail set.

Proposed kernel:
---

$ uname -rv
5.8.0-30-generic #32-Ubuntu SMP Mon Nov 9 21:02:16 UTC 2020

$ VERSION='5.8.0-30-generic #32-Ubuntu'

Number of runs:

$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
12
$ RUNS=12

Consistent fail set:

$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
     12 btrfs/153
     12 btrfs/208
     12 btrfs/213
     12 btrfs/219
     12 btrfs/220
     12 btrfs/221
     12 btrfs/222
     12 generic/133 << not a regression; failed 11/12 in original kernel (comment #9)
     12 generic/260
     12 generic/286
     12 generic/301
     12 generic/465
     12 generic/610

Flaky fail set:

$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
      1 btrfs/020
      1 generic/298
      9 generic/175
     10 generic/475
     11 btrfs/010
     11 generic/074
     11 generic/142
     11 generic/143

[Mauricio Faria de Oliveira (mfo)]

Verification done for focal-proposed.
No regressions in the consistent/flaky fail sets.

Proposed kernel:
---

$ uname -rv
5.4.0-55-generic #61-Ubuntu SMP Mon Nov 9 20:45:34 UTC 2020

$ VERSION='5.4.0-55-generic #61-Ubuntu'

Number of runs:

$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
11
$ RUNS=11

Consistent fail set:

$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
     11 btrfs/153
     11 btrfs/205
     11 btrfs/213
     11 btrfs/220
     11 btrfs/221
     11 btrfs/222
     11 generic/133 << not a regression; failed 12/13 in original master-next
     11 generic/260
     11 generic/286
     11 generic/301
     11 generic/465
     11 generic/484
     11 generic/610

btrfs/200 fixed.
btrfs/204 fixed.
btrfs/226 fixed.

Flaky fail set:

$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
      1 generic/297
      1 generic/371
      3 generic/166
      9 generic/175
     10 btrfs/010
     10 generic/074
     10 generic/142
     10 generic/143
     10 generic/475

[Mauricio Faria de Oliveira (mfo)]

Verification done for focal-proposed.
No regressions in the consistent/flaky fail sets.

The flaky fail set had a long tail (tests that failed just a few times),
but it can be isolated to another btrfs commit, and it doesn't seem to
happen often. (e.g. not seen on focal-proposed, but seen on focal orig
kernel on master-next, and both have the commit.)
Thus not a general/permanent issue. More details in the next comment.

Proposed kernel:
---

$ uname -rv
4.15.0-125-generic #128-Ubuntu SMP Mon Nov 9 20:45:41 UTC 2020

$ VERSION='4.15.0-125-generic #128-Ubuntu'

Number of runs:

$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep -c Failures:
14
$ RUNS=14

Consistent fail set:

$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w $RUNS
     14 btrfs/186
     14 btrfs/193
     14 btrfs/200
     14 btrfs/202
     14 btrfs/204
     14 btrfs/205
     14 btrfs/214
     14 btrfs/219
     14 btrfs/220
     14 btrfs/221
     14 btrfs/222
     14 generic/260
     14 generic/286
     14 generic/301
     14 generic/371
     14 generic/405
     14 generic/407
     14 generic/465
     14 generic/480
     14 generic/488
     14 generic/502
     14 generic/513
     14 generic/526
     14 generic/527
     14 generic/546
     14 generic/562
     14 generic/565

Flaky fail set:

$ grep -h -e ^Fail -e ^PLATFORM xfstests.log.* | grep -A1 "$VERSION" | grep ^Failures: | cut -d' ' -f2- | tr ' ' '\n' | sort | uniq -c | sort | grep -w -v $RUNS
      1 btrfs/001
      1 btrfs/147
      1 btrfs/148
      1 btrfs/149
      1 btrfs/150
      1 btrfs/151
      1 btrfs/152
      1 btrfs/153
      1 btrfs/155
      1 btrfs/156
      1 btrfs/159
      1 btrfs/164
      1 btrfs/165
      1 btrfs/166
      1 btrfs/168
      1 btrfs/169
      1 btrfs/170
      1 btrfs/171
      1 btrfs/172
      1 btrfs/173
      1 btrfs/174
      1 btrfs/177
      1 btrfs/178
      1 btrfs/179
      1 btrfs/180
      1 btrfs/181
      1 btrfs/182
      1 btrfs/183
      1 btrfs/184
      1 btrfs/187
      1 btrfs/188
      1 btrfs/190
      1 btrfs/196
      1 btrfs/197
      1 btrfs/199
      1 btrfs/206
      1 btrfs/207
      1 btrfs/208
      1 btrfs/213
      1 btrfs/217
      1 generic/003
      1 generic/017
      1 generic/031
      1 generic/033
      1 generic/037
      1 generic/038
      1 generic/039
      1 generic/040
      1 generic/041
      1 generic/043
      1 generic/044
      1 generic/045
      1 generic/046
      1 generic/047
      1 generic/048
      1 generic/049
      1 generic/050
      1 generic/051
      1 generic/052
      1 generic/053
      1 generic/054
      1 generic/055
      1 generic/057
      1 generic/059
      1 generic/062
      1 generic/064
      1 generic/065
      1 generic/066
      1 generic/067
      1 generic/068
      1 generic/069
      1 generic/071
      1 generic/073
      1 generic/076
      1 generic/077
      1 generic/079
      1 generic/081
      1 generic/082
      1 generic/083
      1 generic/084
      1 generic/090
      1 generic/1...

[Mauricio Faria de Oliveira (mfo)]

In comment #17, read:
Verification done for BIONIC-proposed. (see 4.15 kernel.)

[Mauricio Faria de Oliveira (mfo)]

Explanatory note on bionic-proposed (see comment #17):

The flaky fail set had a long tail (tests that failed just a few times).

Symptom:
---

The logs show the failure cause is output mismatch, because '/target'
failed to umount (busy), and it was logged; e.g.,

 btrfs/027 [failed, exit status 1]- output mismatch (see /home/ubuntu/xfstests-dev/results//btrfs/027.out.bad)
     --- tests/btrfs/027.out 2020-10-26 22:23:54.626341276 +0000
     +++ /home/ubuntu/xfstests-dev/results//btrfs/027.out.bad 2020-11-17 06:34:12.623548481 +0000
     @@ -1,2 +1,3 @@
      QA output created by 027
     -Silence is golden
     +umount: /mnt/scratch: target is busy.
     +failed to unmount /dev/loop2
     ...

The number of failed tests increases with the number of umount failures.
Comparing both numbers per log file: (line breaks added for clarity.)

 $ (grep ^Failed xfstests.log.2020-11-1*; \
    grep -c 'umount: /mnt/scratch: target is busy' xfstests.log.2020-11-1* | grep -v :0) \
      | sort
 xfstests.log.2020-11-16-21-03-32:Failed 31 of 823 tests // 30ish is normal failure count
 xfstests.log.2020-11-16-22-16-38:Failed 34 of 823 tests
 xfstests.log.2020-11-16-23-25-56:Failed 34 of 823 tests
 xfstests.log.2020-11-17-00-36-27:Failed 36 of 823 tests
 xfstests.log.2020-11-17-01-44-04:Failed 37 of 823 tests
 xfstests.log.2020-11-17-02-52-41:Failed 37 of 823 tests // things go bad after this. umount hung here.

 xfstests.log.2020-11-17-04-07-11:15
 xfstests.log.2020-11-17-04-07-11:Failed 68 of 823 tests // umount hung here too.

 xfstests.log.2020-11-17-05-21-34:76
 xfstests.log.2020-11-17-05-21-34:Failed 158 of 823 tests

 xfstests.log.2020-11-17-06-29-11:148
 xfstests.log.2020-11-17-06-29-11:Failed 278 of 823 tests

 xfstests.log.2020-11-17-07-29-57:31
 xfstests.log.2020-11-17-07-29-57:Failed 104 of 823 tests

 xfstests.log.2020-11-17-08-37-16:33
 xfstests.log.2020-11-17-08-37-16:Failed 87 of 823 tests

 xfstests.log.2020-11-17-09-50-43:36
 xfstests.log.2020-11-17-09-50-43:Failed 101 of 823 tests

 xfstests.log.2020-11-17-11-04-16:43
 xfstests.log.2020-11-17-11-04-16:Failed 111 of 823 tests

 xfstests.log.2020-11-17-12-08-14:68
 xfstests.log.2020-11-17-12-08-14:Failed 143 of 823 tests

 xfstests.log.2020-11-17-13-14-01:93
 xfstests.log.2020-11-17-13-14-01:Failed 200 of 823 tests

And the kernel log files show that 'umount' hung in the timestamps where the increase started.

 $ sudo grep umount kern.log.for.2020-11-1*
 kern.log.for.2020-11-17-02-52-41:Nov 17 03:52:55 mfo-s390x-bionic kernel: [24748.572259] CPU: 4 PID: 10862 Comm: umount Tainted: G W 4.15.0-125-generic #128-Ubuntu
 kern.log.for.2020-11-17-02-52-41:Nov 17 03:53:10 mfo-s390x-bionic kernel: [24762.822089] CPU: 4 PID: 11705 Comm: umount Tainted: G W 4.15.0-125-generic #128-Ubuntu
 kern.log.for.2020-11-17-02-52-41:Nov 17 03:53:16 mfo-s390x-bionic kernel: [24768.881926] CPU: 4 PID: 12110 Comm: umount Tainted: G W 4.15.0-125-generic #128-Ubuntu

 kern.log.for.2020-11-17-04-07-11:Nov 17 03:52:55 mfo-s390x-bionic kernel: [24748.572259] CPU: 4 PID: 10862 Comm: umount Tainted: G W 4.15.0-125-generic #128-Ubuntu
 kern.log.for.2020-11-1...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.4.0-56.62

---------------
linux (5.4.0-56.62) focal; urgency=medium

  * focal/linux: 5.4.0-56.62 -proposed tracker (LP: #1905300)

  * CVE-2020-4788
    - selftests/powerpc: rfi_flush: disable entry flush if present
    - powerpc/64s: flush L1D on kernel entry
    - powerpc/64s: flush L1D after user accesses
    - selftests/powerpc: entry flush test

linux (5.4.0-55.61) focal; urgency=medium

  * focal/linux: 5.4.0-55.61 -proposed tracker (LP: #1903175)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * CVE-2020-14351
    - perf/core: Fix race in the perf_mmap_close() function

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout
    - dm raid: fix discard limits for raid1 and raid10
    - dm raid: remove unnecessary discard limits for raid10

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: drop unnecessary offset_in_page in extent buffer helpers
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Ethernet no link lights after reboot (Intel i225-v 2.5G) (LP: #1902578)
    - igc: Add PHY power management control

  * Undetected Data corruption in MPI workloads that use VSX for reductions on
    POWER9 DD2.1 systems (LP: #1902694)
    - powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
    - selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
      workaround

  * [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
    - s390: nvme ipl
    - s390: nvme reipl
    - s390/ipl: support NVMe IPL kernel parameters

  * uvcvideo: add mapping for HEVC payloads (LP: #1895803)
    - media: uvcvideo: Add mapping for HEVC payloads

  * Focal update: v5.4.73 upstream stable release (LP: #1902115)
    - ibmveth: Switch order of ibmveth_helper calls.
    - ibmveth: Identify ingress large send packets.
    - ipv4: Restore flowi4_oif update before call to xfrm_lookup_route
    - mlx4: handle non-napi callers to napi_poll
    - net: fec: Fix phy_device lookup for phy_reset_after_clk_enable()
    - net: fec: Fix PHY init after phy_reset_after_clk_enable()
    - net: fix pos incrementment in ipv6_route_seq_next
    - net/smc: fix valid DMBE buffer sizes
    - net...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.8.0-31.33

---------------
linux (5.8.0-31.33) groovy; urgency=medium

  * groovy/linux: 5.8.0-31.33 -proposed tracker (LP: #1905299)

  * Groovy 5.8 kernel hangs on boot on CPUs with eLLC (LP: #1903397)
    - drm/i915: Mark ininitial fb obj as WT on eLLC machines to avoid rcu lockup
      during fbdev init

  * CVE-2020-4788
    - selftests/powerpc: rfi_flush: disable entry flush if present
    - powerpc/64s: flush L1D on kernel entry
    - powerpc/64s: flush L1D after user accesses
    - selftests/powerpc: entry flush test

linux (5.8.0-30.32) groovy; urgency=medium

  * groovy/linux: 5.8.0-30.32 -proposed tracker (LP: #1903194)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout
    - dm raid: fix discard limits for raid1 and raid10
    - dm raid: remove unnecessary discard limits for raid10

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Tiger Lake PMC core driver fixes (LP: #1899883)
    - platform/x86: intel_pmc_core: update TGL's LPM0 reg bit map name
    - platform/x86: intel_pmc_core: fix bound check in pmc_core_mphy_pg_show()
    - platform/x86: pmc_core: Use descriptive names for LPM registers
    - platform/x86: intel_pmc_core: Fix TigerLake power gating status map
    - platform/x86: intel_pmc_core: Fix the slp_s0 counter displayed value

  * drm/i915/dp_mst - System would hang during the boot up. (LP: #1902469)
    - Revert "UBUNTU: SAUCE: drm/i915/display: Fix null deref in
      intel_psr_atomic_check()"
    - drm/i915: Fix encoder lookup during PSR atomic check

  * Undetected Data corruption in MPI workloads that use VSX for reductions on
    POWER9 DD2.1 systems (LP: #1902694)
    - powerpc: Fix undetected data corruption with P9N DD2.1 VSX CI load emulation
    - selftests/powerpc: Make alignment handler test P9N DD2.1 vector CI load
      workaround

  * [20.04 FEAT] Support/enhancement of NVMe IPL (LP: #1902179)
    - s390/ipl: support NVMe IPL kernel parameters

  * uvcvideo: add mapping for HEVC payloads (LP: #1895803)
    - media: uvcvideo: Add mapping for HEVC payloads

  * risc-v 5.8 ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.15.0-126.129

---------------
linux (4.15.0-126.129) bionic; urgency=medium

  * bionic/linux: 4.15.0-126.129 -proposed tracker (LP: #1905305)

  * CVE-2020-4788
    - SAUCE: powerpc/64s: Define MASKABLE_RELON_EXCEPTION_PSERIES_OOL
    - SAUCE: powerpc/64s: move some exception handlers out of line
    - powerpc/64s: flush L1D on kernel entry
    - SAUCE: powerpc: Add a framework for user access tracking
    - powerpc: Implement user_access_begin and friends
    - powerpc: Fix __clear_user() with KUAP enabled
    - powerpc/uaccess: Evaluate macro arguments once, before user access is
      allowed
    - powerpc/64s: flush L1D after user accesses

linux (4.15.0-125.128) bionic; urgency=medium

  * bionic/linux: 4.15.0-125.128 -proposed tracker (LP: #1903137)

  * Update kernel packaging to support forward porting kernels (LP: #1902957)
    - [Debian] Update for leader included in BACKPORT_SUFFIX

  * Avoid double newline when running insertchanges (LP: #1903293)
    - [Packaging] insertchanges: avoid double newline

  * EFI: Fails when BootCurrent entry does not exist (LP: #1899993)
    - efivarfs: Replace invalid slashes with exclamation marks in dentries.

  * CVE-2020-14351
    - perf/core: Fix race in the perf_mmap_close() function

  * raid10: Block discard is very slow, causing severe delays for mkfs and
    fstrim operations (LP: #1896578)
    - md: add md_submit_discard_bio() for submitting discard bio
    - md/raid10: extend r10bio devs to raid disks
    - md/raid10: pull codes that wait for blocked dev into one function
    - md/raid10: improve raid10 discard request
    - md/raid10: improve discard request for far layout

  * Bionic: btrfs: kernel BUG at /build/linux-
    eTBZpZ/linux-4.15.0/fs/btrfs/ctree.c:3233! (LP: #1902254)
    - btrfs: use offset_in_page instead of open-coding it
    - btrfs: use BUG() instead of BUG_ON(1)
    - btrfs: drop unnecessary offset_in_page in extent buffer helpers
    - btrfs: extent_io: do extra check for extent buffer read write functions
    - btrfs: extent-tree: kill BUG_ON() in __btrfs_free_extent()
    - btrfs: extent-tree: kill the BUG_ON() in insert_inline_extent_backref()
    - btrfs: ctree: check key order before merging tree blocks

  * Bionic update: upstream stable patchset 2020-11-04 (LP: #1902943)
    - USB: gadget: f_ncm: Fix NDP16 datagram validation
    - gpio: tc35894: fix up tc35894 interrupt configuration
    - vsock/virtio: use RCU to avoid use-after-free on the_virtio_vsock
    - vsock/virtio: stop workers during the .remove()
    - vsock/virtio: add transport parameter to the
      virtio_transport_reset_no_sock()
    - net: virtio_vsock: Enhance connection semantics
    - Input: i8042 - add nopnp quirk for Acer Aspire 5 A515
    - ftrace: Move RCU is watching check after recursion check
    - drm/amdgpu: restore proper ref count in amdgpu_display_crtc_set_config
    - drivers/net/wan/hdlc_fr: Add needed_headroom for PVC devices
    - drm/sun4i: mixer: Extend regmap max_register
    - net: dec: de2104x: Increase receive ring size for Tulip
    - rndis_host: increase sleep time in the query-response loop
    - nvme-core: get/put ctrl ...

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2020-12-02 03:36 EDT-------
IBM Bugzilla status-> closed, Fix Released by Canonical

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.8.0-36.40+21.04.1

---------------
linux (5.8.0-36.40+21.04.1) hirsute; urgency=medium

  * Packaging resync (LP: #1786013)
    - update dkms package versions

  [ Ubuntu: 5.8.0-36.40 ]

  * debian/scripts/file-downloader does not handle positive failures correctly
    (LP: #1878897)
    - [Packaging] file-downloader not handling positive failures correctly

  [ Ubuntu: 5.8.0-35.39 ]

  * Packaging resync (LP: #1786013)
    - update dkms package versions
  * CVE-2021-1052 // CVE-2021-1053
    - [Packaging] NVIDIA -- Add the NVIDIA 460 driver

 -- Kleber Sacilotto de Souza <email address hidden> Thu, 07 Jan 2021 11:57:30 +0100