Activity log for bug #1852122

Date Who What changed Old value New value Message
2019-11-11 19:39:05 Iain Lane bug added bug
2019-11-14 20:14:46 Andreas Hasenack attachment added Screenshot from 2019-11-14 17-13-34.png https://bugs.launchpad.net/ubuntu/+source/ocfs2-tools/+bug/1852122/+attachment/5305453/+files/Screenshot%20from%202019-11-14%2017-13-34.png
2019-11-14 20:14:53 Andreas Hasenack ocfs2-tools (Ubuntu): status New Confirmed
2019-11-14 20:14:58 Andreas Hasenack ocfs2-tools (Ubuntu): status Confirmed Triaged
2019-11-14 20:15:04 Andreas Hasenack ocfs2-tools (Ubuntu): importance Undecided Medium
2019-11-14 20:15:11 Andreas Hasenack bug added subscriber Ubuntu Server
2019-11-14 21:10:38 Rafael David Tinoco bug added subscriber Ubuntu Server High Availability
2019-12-09 13:31:22 Rafael David Tinoco bug added subscriber Ubuntu Kernel Bot
2019-12-09 13:31:30 Rafael David Tinoco ocfs2-tools (Ubuntu): status Triaged Confirmed
2019-12-09 13:33:13 Rafael David Tinoco ocfs2-tools (Ubuntu): assignee Rafael David Tinoco (rafaeldtinoco)
2019-12-09 13:33:24 Rafael David Tinoco removed subscriber Ubuntu Kernel Bot
2020-01-10 05:07:05 Rafael David Tinoco summary ocfs2-tools autopkgtest is causing kernel panics on ppc64el ocfs2-tools autopkgtest is causing kernel panics on service shutdown
2020-01-10 05:07:38 Rafael David Tinoco attachment added ocfs2_reproducer.sh https://bugs.launchpad.net/ubuntu/+source/ocfs2-tools/+bug/1852122/+attachment/5319035/+files/ocfs2_reproducer.sh
2020-01-10 05:07:55 Rafael David Tinoco ocfs2-tools (Ubuntu): status Confirmed In Progress
2020-01-10 05:08:00 Rafael David Tinoco bug watch added https://github.com/markfasheh/ocfs2-tools/issues/45
2020-01-10 05:08:00 Rafael David Tinoco bug task added ocfs2-tools
2020-01-10 05:19:07 Rafael David Tinoco summary ocfs2-tools autopkgtest is causing kernel panics on service shutdown ocfs2-tools is causing kernel panics in Ubuntu Focal (Ubuntu-5.4.0-9.12)
2020-01-10 05:19:26 Rafael David Tinoco bug task added linux (Ubuntu)
2020-01-10 05:19:32 Rafael David Tinoco linux (Ubuntu): status New In Progress
2020-01-10 05:19:38 Rafael David Tinoco linux (Ubuntu): importance Undecided Medium
2020-01-10 05:19:47 Rafael David Tinoco nominated for series Ubuntu Focal
2020-01-10 05:19:47 Rafael David Tinoco bug task added ocfs2-tools (Ubuntu Focal)
2020-01-10 05:19:47 Rafael David Tinoco bug task added linux (Ubuntu Focal)
2020-01-10 05:22:05 Rafael David Tinoco nominated for series Ubuntu Eoan
2020-01-10 05:22:05 Rafael David Tinoco bug task added ocfs2-tools (Ubuntu Eoan)
2020-01-10 05:22:05 Rafael David Tinoco bug task added linux (Ubuntu Eoan)
2020-01-10 05:22:15 Rafael David Tinoco linux (Ubuntu Eoan): status New In Progress
2020-01-10 05:22:16 Rafael David Tinoco ocfs2-tools (Ubuntu Eoan): status New In Progress
2020-01-10 05:22:19 Rafael David Tinoco ocfs2-tools (Ubuntu Eoan): importance Undecided Medium
2020-01-10 05:22:20 Rafael David Tinoco linux (Ubuntu Eoan): importance Undecided Medium
2020-01-10 05:22:22 Rafael David Tinoco ocfs2-tools (Ubuntu Eoan): assignee Rafael David Tinoco (rafaeldtinoco)
2020-01-10 05:28:15 Bug Watch Updater ocfs2-tools: status Unknown New
2020-01-10 13:20:58 Andreas Hasenack tags update-excuse
2020-01-13 07:27:28 Bug Watch Updater ocfs2-tools: status New Fix Released
2020-01-13 19:06:08 Rafael David Tinoco ocfs2-tools (Ubuntu Eoan): status In Progress Invalid
2020-01-13 19:06:10 Rafael David Tinoco ocfs2-tools (Ubuntu Focal): status In Progress Invalid
2020-01-13 19:58:40 Rafael David Tinoco description I noticed the tests for ocfs2-tools/1.8.6-1ubuntu1 were constantly retrying themselves. It's a feature we have so that transient / occasional failures are auto-retried, but it's misfiring here because we're not detecting that it's a consistent failure. That particular bug is fixed, but it means that ocfs2-tools is failing on ppc64el. Here's the important part of the log, full output attached. [ 85.605738] BUG: Unable to handle kernel data access at 0x01744098 [ 85.605850] Faulting instruction address: 0xc000000000e81168 [ 85.605901] Oops: Kernel access of bad area, sig: 11 [#1] [ 85.605970] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries [ 85.606029] Modules linked in: ocfs2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue iptable_mangle xt_TCPMSS xt_tcpudp bpfilter dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq libcrc32c crc32c_vpmsum virtio_net virtio_blk net_failover failover [ 85.606291] CPU: 0 PID: 1 Comm: systemd Not tainted 5.3.0-18-generic #19-Ubuntu [ 85.606350] NIP: c000000000e81168 LR: c00000000054f240 CTR: 0000000000000000 [ 85.606410] REGS: c00000005a3e3700 TRAP: 0300 Not tainted (5.3.0-18-generic) [ 85.606469] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 28024448 XER: 00000000 [ 85.606531] CFAR: 0000701f9806f638 DAR: 0000000001744098 DSISR: 40000000 IRQMASK: 0 [ 85.606531] GPR00: 0000000000007374 c00000005a3e3990 c0000000019c9100 c00000004fe462a8 [ 85.606531] GPR04: c00000005856d840 000000000000000e 0000000074656772 c00000004fe4a568 [ 85.606531] GPR08: 0000000000000000 c000000058568004 0000000001744090 0000000000000000 [ 85.606531] GPR12: 00000000e8086002 c000000001d60000 00007fffddd522d0 0000000000000000 [ 85.606531] GPR16: 0000000000000000 0000000000000000 0000000000000000 c00000000755e07c [ 85.606531] GPR20: c0000000598caca8 c00000005a3e3a58 0000000000000000 c000000058292f00 [ 85.606531] GPR24: c000000000eea710 0000000000000000 c00000005856d840 c00000000755e074 [ 85.606531] GPR28: 000000006518907d c00000005a3e3a68 c00000004fe4b160 00000000027c47b6 [ 85.607079] NIP [c000000000e81168] rb_insert_color+0x18/0x1c0 [ 85.607137] LR [c00000000054f240] ext4_htree_store_dirent+0x140/0x1c0 [ 85.607186] Call Trace: [ 85.607208] [c00000005a3e3990] [c00000000054f158] ext4_htree_store_dirent+0x58/0x1c0 (unreliable) [ 85.607279] [c00000005a3e39e0] [c000000000594cd8] htree_dirblock_to_tree+0x1b8/0x380 [ 85.607340] [c00000005a3e3b00] [c0000000005962c0] ext4_htree_fill_tree+0xc0/0x3f0 [ 85.607401] [c00000005a3e3c00] [c00000000054ebe4] ext4_readdir+0x814/0xce0 [ 85.607459] [c00000005a3e3d40] [c000000000472d6c] iterate_dir+0x1fc/0x280 [ 85.607511] [c00000005a3e3d90] [c0000000004746f0] ksys_getdents64+0xa0/0x1f0 [ 85.607572] [c00000005a3e3e00] [c000000000474868] sys_getdents64+0x28/0x130 [ 85.607622] [c00000005a3e3e20] [c00000000000b388] system_call+0x5c/0x70 [ 85.607672] Instruction dump: [ 85.607703] 4082ffe8 4e800020 38600000 4e800020 60000000 60000000 e9230000 2c290000 [ 85.607764] 4182018c e9490000 71480001 4c820020 <e90a0008> 7c284840 2fa80000 4182006c [ 85.607827] ---[ end trace cfc53af0f8d62cef ]--- [ 85.610600] [ 86.611522] BUG: Unable to handle kernel data access at 0xc000030058567eff [ 86.611604] Faulting instruction address: 0xc000000000403aa8 [ 86.611656] Oops: Kernel access of bad area, sig: 11 [#2] [ 86.611697] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries [ 86.611748] Modules linked in: ocfs2 quota_tr [Impact] * Umounts to OCFS2 filesystems will cause a kernel crash in any kernel containing commit e581595ea29c (v5.3-rc1) and not containing its fix b73eba2a867e (v5.5-rc5). [Test Case] * ocfs2_reproducer.sh (attached to this bug) [Regression Potential] * it could cause ocfs2 issues, so I guess its a low impact considering amount of users. * Its a straightforward fix identified by upstream developer and a clean cherry-pick for us. [Other Info] * Original description: I noticed the tests for ocfs2-tools/1.8.6-1ubuntu1 were constantly retrying themselves. It's a feature we have so that transient / occasional failures are auto-retried, but it's misfiring here because we're not detecting that it's a consistent failure. That particular bug is fixed, but it means that ocfs2-tools is failing on ppc64el. Here's the important part of the log, full output attached. [ 85.605738] BUG: Unable to handle kernel data access at 0x01744098                                                                                                                  [ 85.605850] Faulting instruction address: 0xc000000000e81168                                                                                                                  [ 85.605901] Oops: Kernel access of bad area, sig: 11 [#1]                                                                                                                  [ 85.605970] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries                                                                                                                  [ 85.606029] Modules linked in: ocfs2 quota_tree ocfs2_dlmfs ocfs2_stack_o2cb ocfs2_dlm ocfs2_nodemanager ocfs2_stackglue iptable_mangle xt_TCPMSS xt_tcpudp bpfilter dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua vmx_crypto crct10dif_vpmsum sch_fq_codel ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq libcrc32c crc32c_vpmsum virtio_net virtio_blk net_failover failover                                                                                                                  [ 85.606291] CPU: 0 PID: 1 Comm: systemd Not tainted 5.3.0-18-generic #19-Ubuntu                                                                                                                  [ 85.606350] NIP: c000000000e81168 LR: c00000000054f240 CTR: 0000000000000000                                                                                                                  [ 85.606410] REGS: c00000005a3e3700 TRAP: 0300 Not tainted (5.3.0-18-generic)                                                                                                                  [ 85.606469] MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 28024448 XER: 00000000                                                                                                                  [ 85.606531] CFAR: 0000701f9806f638 DAR: 0000000001744098 DSISR: 40000000 IRQMASK: 0                                                                                                                  [ 85.606531] GPR00: 0000000000007374 c00000005a3e3990 c0000000019c9100 c00000004fe462a8                                                                                                                  [ 85.606531] GPR04: c00000005856d840 000000000000000e 0000000074656772 c00000004fe4a568                                                                                                                  [ 85.606531] GPR08: 0000000000000000 c000000058568004 0000000001744090 0000000000000000                                                                                                                  [ 85.606531] GPR12: 00000000e8086002 c000000001d60000 00007fffddd522d0 0000000000000000                                                                                                                  [ 85.606531] GPR16: 0000000000000000 0000000000000000 0000000000000000 c00000000755e07c                                                                                                                  [ 85.606531] GPR20: c0000000598caca8 c00000005a3e3a58 0000000000000000 c000000058292f00                                                                                                                  [ 85.606531] GPR24: c000000000eea710 0000000000000000 c00000005856d840 c00000000755e074                                                                                                                  [ 85.606531] GPR28: 000000006518907d c00000005a3e3a68 c00000004fe4b160 00000000027c47b6                                                                                                                  [ 85.607079] NIP [c000000000e81168] rb_insert_color+0x18/0x1c0                                                                                                                  [ 85.607137] LR [c00000000054f240] ext4_htree_store_dirent+0x140/0x1c0                                                                                                                  [ 85.607186] Call Trace:                                                                                                                  [ 85.607208] [c00000005a3e3990] [c00000000054f158] ext4_htree_store_dirent+0x58/0x1c0 (unreliable)                                                                                                                  [ 85.607279] [c00000005a3e39e0] [c000000000594cd8] htree_dirblock_to_tree+0x1b8/0x380                                                                                                                  [ 85.607340] [c00000005a3e3b00] [c0000000005962c0] ext4_htree_fill_tree+0xc0/0x3f0                                                                                                                  [ 85.607401] [c00000005a3e3c00] [c00000000054ebe4] ext4_readdir+0x814/0xce0                                                                                                                  [ 85.607459] [c00000005a3e3d40] [c000000000472d6c] iterate_dir+0x1fc/0x280                                                                                                                  [ 85.607511] [c00000005a3e3d90] [c0000000004746f0] ksys_getdents64+0xa0/0x1f0                                                                                                                  [ 85.607572] [c00000005a3e3e00] [c000000000474868] sys_getdents64+0x28/0x130                                                                                                                  [ 85.607622] [c00000005a3e3e20] [c00000000000b388] system_call+0x5c/0x70                                                                                                                  [ 85.607672] Instruction dump:                                                                                                                  [ 85.607703] 4082ffe8 4e800020 38600000 4e800020 60000000 60000000 e9230000 2c290000                                                                                                                  [ 85.607764] 4182018c e9490000 71480001 4c820020 <e90a0008> 7c284840 2fa80000 4182006c                                                                                                                  [ 85.607827] ---[ end trace cfc53af0f8d62cef ]---                                                                                                                  [ 85.610600]                                                                                                                  [ 86.611522] BUG: Unable to handle kernel data access at 0xc000030058567eff                                                                                                                  [ 86.611604] Faulting instruction address: 0xc000000000403aa8                                                                                                                  [ 86.611656] Oops: Kernel access of bad area, sig: 11 [#2]                                                                                                                  [ 86.611697] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries                                                                                                                  [ 86.611748] Modules linked in: ocfs2 quota_tr
2020-01-23 22:37:43 Rafael David Tinoco linux (Ubuntu Focal): status In Progress Fix Released
2020-01-29 16:04:18 Marcelo Cerri linux (Ubuntu Eoan): status In Progress Fix Committed
2020-02-03 22:43:26 Ubuntu Kernel Bot tags update-excuse update-excuse verification-needed-eoan
2020-02-12 11:07:25 Rafael David Tinoco tags update-excuse verification-needed-eoan update-excuse verification-done verification-done-eoan
2020-02-17 10:23:38 Launchpad Janitor linux (Ubuntu Eoan): status Fix Committed Fix Released
2020-02-17 10:23:38 Launchpad Janitor cve linked 2019-19050
2020-02-17 10:23:38 Launchpad Janitor cve linked 2019-19077
2020-02-17 10:23:38 Launchpad Janitor cve linked 2019-19078
2020-02-17 10:23:38 Launchpad Janitor cve linked 2019-19082
2020-02-17 10:23:38 Launchpad Janitor cve linked 2019-19332
2020-02-17 10:23:38 Launchpad Janitor cve linked 2019-19965