Activity log for bug #1799497

Date Who What changed Old value New value Message
2018-10-23 16:03:10 Stéphane Graber bug added bug
2018-10-23 16:10:44 Stefan Bader bug added subscriber Stefan Bader
2018-10-23 16:28:16 Joseph Salisbury linux (Ubuntu): importance Undecided High
2018-10-23 16:28:25 Joseph Salisbury nominated for series Ubuntu Bionic
2018-10-23 16:28:25 Joseph Salisbury bug task added linux (Ubuntu Bionic)
2018-10-23 16:28:33 Joseph Salisbury linux (Ubuntu Bionic): importance Undecided High
2018-10-23 16:29:00 Joseph Salisbury tags bionic kernel-key
2018-10-23 16:30:05 Ubuntu Kernel Bot linux (Ubuntu): status New Incomplete
2018-10-23 16:30:07 Ubuntu Kernel Bot linux (Ubuntu Bionic): status New Incomplete
2018-10-23 18:02:01 Stéphane Graber tags bionic kernel-key apport-collected bionic kernel-key
2018-10-23 18:02:02 Stéphane Graber description My main server has been running into hard lockups about once a week ever since I switched to the 4.15 Ubuntu 18.04 kernel. When this happens, nothing is printed to the console, it's effectively stuck showing a login prompt. The system is running with panic=1 on the cmdline but isn't rebooting so the kernel isn't even processing this as a kernel panic. As this felt like a potential hardware issue, I had my hosting provider give me a completely different system, different motherboard, different CPU, different RAM and different storage, I installed that system on 18.04 and moved my data over, a week later, I hit the issue again. We've since also had a LXD user reporting similar symptoms here also on varying hardware: https://github.com/lxc/lxd/issues/5197 My system doesn't have a lot of memory pressure with about 50% of free memory: root@vorash:~# free -m total used free shared buff/cache available Mem: 31819 17574 402 513 13842 13292 Swap: 15909 2687 13222 I will now try to increase console logging as much as possible on the system in the hopes that next time it hangs we can get a better idea of what happened but I'm not too hopeful given the complete silence on the console when this occurs. System is currently on: Linux vorash 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux But I've seen this since the GA kernel on 4.15 so it's not a recent regression. My main server has been running into hard lockups about once a week ever since I switched to the 4.15 Ubuntu 18.04 kernel. When this happens, nothing is printed to the console, it's effectively stuck showing a login prompt. The system is running with panic=1 on the cmdline but isn't rebooting so the kernel isn't even processing this as a kernel panic. As this felt like a potential hardware issue, I had my hosting provider give me a completely different system, different motherboard, different CPU, different RAM and different storage, I installed that system on 18.04 and moved my data over, a week later, I hit the issue again. We've since also had a LXD user reporting similar symptoms here also on varying hardware: https://github.com/lxc/lxd/issues/5197 My system doesn't have a lot of memory pressure with about 50% of free memory: root@vorash:~# free -m total used free shared buff/cache available Mem: 31819 17574 402 513 13842 13292 Swap: 15909 2687 13222 I will now try to increase console logging as much as possible on the system in the hopes that next time it hangs we can get a better idea of what happened but I'm not too hopeful given the complete silence on the console when this occurs. System is currently on: Linux vorash 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux But I've seen this since the GA kernel on 4.15 so it's not a recent regression. --- ProblemType: Bug AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 23 16:12 seq crw-rw---- 1 root audio 116, 33 Oct 23 16:12 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.9-0ubuntu7.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Cannot stat file /proc/22822/fd/10: Permission denied Cannot stat file /proc/22831/fd/10: Permission denied DistroRelease: Ubuntu 18.04 HibernationDevice: RESUME=none CRYPTSETUP=n IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' Lsusb: Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 002: ID 046b:ff10 American Megatrends, Inc. Virtual Keyboard and Mouse Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Intel Corporation S1200SP NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 mgadrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-38-generic root=UUID=575c878a-0be6-4806-9c83-28f67aedea65 ro biosdevname=0 net.ifnames=0 panic=1 verbose console=tty0 console=ttyS0,115200n8 ProcVersionSignature: Ubuntu 4.15.0-38.41-generic 4.15.18 RelatedPackageVersions: linux-restricted-modules-4.15.0-38-generic N/A linux-backports-modules-4.15.0-38-generic N/A linux-firmware 1.173.1 RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' Tags: bionic Uname: Linux 4.15.0-38-generic x86_64 UnreportableReason: This report is about a package that is not installed. UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: False dmi.bios.date: 01/25/2018 dmi.bios.vendor: Intel Corporation dmi.bios.version: S1200SP.86B.03.01.1029.012520180838 dmi.board.asset.tag: Base Board Asset Tag dmi.board.name: S1200SP dmi.board.vendor: Intel Corporation dmi.board.version: H57532-271 dmi.chassis.asset.tag: .................... dmi.chassis.type: 23 dmi.chassis.vendor: ............................... dmi.chassis.version: .................. dmi.modalias: dmi:bvnIntelCorporation:bvrS1200SP.86B.03.01.1029.012520180838:bd01/25/2018:svnIntelCorporation:pnS1200SP:pvr....................:rvnIntelCorporation:rnS1200SP:rvrH57532-271:cvn...............................:ct23:cvr..................: dmi.product.family: Family dmi.product.name: S1200SP dmi.product.version: .................... dmi.sys.vendor: Intel Corporation
2018-10-23 18:02:03 Stéphane Graber attachment added CRDA.txt https://bugs.launchpad.net/bugs/1799497/+attachment/5204632/+files/CRDA.txt
2018-10-23 18:02:04 Stéphane Graber attachment added CurrentDmesg.txt https://bugs.launchpad.net/bugs/1799497/+attachment/5204633/+files/CurrentDmesg.txt
2018-10-23 18:02:07 Stéphane Graber attachment added Lspci.txt https://bugs.launchpad.net/bugs/1799497/+attachment/5204634/+files/Lspci.txt
2018-10-23 18:02:08 Stéphane Graber attachment added ProcCpuinfoMinimal.txt https://bugs.launchpad.net/bugs/1799497/+attachment/5204635/+files/ProcCpuinfoMinimal.txt
2018-10-23 18:02:09 Stéphane Graber attachment added ProcInterrupts.txt https://bugs.launchpad.net/bugs/1799497/+attachment/5204636/+files/ProcInterrupts.txt
2018-10-23 18:02:10 Stéphane Graber attachment added ProcModules.txt https://bugs.launchpad.net/bugs/1799497/+attachment/5204637/+files/ProcModules.txt
2018-10-23 18:02:13 Stéphane Graber attachment added UdevDb.txt https://bugs.launchpad.net/bugs/1799497/+attachment/5204638/+files/UdevDb.txt
2018-10-23 18:02:15 Stéphane Graber attachment added WifiSyslog.txt https://bugs.launchpad.net/bugs/1799497/+attachment/5204639/+files/WifiSyslog.txt
2018-10-23 18:03:19 Stéphane Graber attachment removed CurrentDmesg.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1799497/+attachment/5204633/+files/CurrentDmesg.txt
2018-10-23 18:03:19 Stéphane Graber attachment removed WifiSyslog.txt https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1799497/+attachment/5204639/+files/WifiSyslog.txt
2018-10-23 19:25:26 Seth Forshee bug added subscriber Seth Forshee
2018-10-24 06:07:38 Luis Rodriguez bug watch added https://github.com/lxc/lxd/issues/5197
2018-10-24 07:15:12 Luis Rodriguez bug added subscriber Luis Rodriguez
2018-10-30 09:42:39 Janåke Rönnblom bug added subscriber Janåke Rönnblom
2018-11-19 15:52:54 Joseph Salisbury tags apport-collected bionic kernel-key apport-collected bionic kernel-da-key
2019-07-24 20:56:44 Brad Figg tags apport-collected bionic kernel-da-key apport-collected bionic cscc kernel-da-key
2019-10-31 01:36:30 David Roberts bug task added zram-config (Ubuntu)
2019-10-31 01:37:13 Launchpad Janitor zram-config (Ubuntu): status New Confirmed
2019-10-31 01:37:13 Launchpad Janitor zram-config (Ubuntu Bionic): status New Confirmed
2019-11-01 11:21:42 David Roberts linux (Ubuntu): status Incomplete Confirmed
2019-11-01 11:21:47 David Roberts linux (Ubuntu Bionic): status Incomplete Confirmed
2019-11-01 11:23:09 David Roberts bug added subscriber David Roberts
2019-12-16 15:28:26 Terry Rudd bug added subscriber Terry Rudd
2020-01-07 20:12:05 Colin Ian King linux (Ubuntu): assignee Colin Ian King (colin-king)
2020-01-09 17:38:47 Colin Ian King linux (Ubuntu): status Confirmed Incomplete
2020-01-09 17:38:53 Colin Ian King zram-config (Ubuntu): status Confirmed Incomplete
2020-02-07 16:23:34 Colin Ian King description My main server has been running into hard lockups about once a week ever since I switched to the 4.15 Ubuntu 18.04 kernel. When this happens, nothing is printed to the console, it's effectively stuck showing a login prompt. The system is running with panic=1 on the cmdline but isn't rebooting so the kernel isn't even processing this as a kernel panic. As this felt like a potential hardware issue, I had my hosting provider give me a completely different system, different motherboard, different CPU, different RAM and different storage, I installed that system on 18.04 and moved my data over, a week later, I hit the issue again. We've since also had a LXD user reporting similar symptoms here also on varying hardware: https://github.com/lxc/lxd/issues/5197 My system doesn't have a lot of memory pressure with about 50% of free memory: root@vorash:~# free -m total used free shared buff/cache available Mem: 31819 17574 402 513 13842 13292 Swap: 15909 2687 13222 I will now try to increase console logging as much as possible on the system in the hopes that next time it hangs we can get a better idea of what happened but I'm not too hopeful given the complete silence on the console when this occurs. System is currently on: Linux vorash 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux But I've seen this since the GA kernel on 4.15 so it's not a recent regression. --- ProblemType: Bug AlsaDevices: total 0 crw-rw---- 1 root audio 116, 1 Oct 23 16:12 seq crw-rw---- 1 root audio 116, 33 Oct 23 16:12 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.9-0ubuntu7.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Cannot stat file /proc/22822/fd/10: Permission denied Cannot stat file /proc/22831/fd/10: Permission denied DistroRelease: Ubuntu 18.04 HibernationDevice: RESUME=none CRYPTSETUP=n IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' Lsusb: Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 002: ID 046b:ff10 American Megatrends, Inc. Virtual Keyboard and Mouse Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Intel Corporation S1200SP NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash ProcFB: 0 mgadrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-38-generic root=UUID=575c878a-0be6-4806-9c83-28f67aedea65 ro biosdevname=0 net.ifnames=0 panic=1 verbose console=tty0 console=ttyS0,115200n8 ProcVersionSignature: Ubuntu 4.15.0-38.41-generic 4.15.18 RelatedPackageVersions: linux-restricted-modules-4.15.0-38-generic N/A linux-backports-modules-4.15.0-38-generic N/A linux-firmware 1.173.1 RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' Tags: bionic Uname: Linux 4.15.0-38-generic x86_64 UnreportableReason: This report is about a package that is not installed. UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: False dmi.bios.date: 01/25/2018 dmi.bios.vendor: Intel Corporation dmi.bios.version: S1200SP.86B.03.01.1029.012520180838 dmi.board.asset.tag: Base Board Asset Tag dmi.board.name: S1200SP dmi.board.vendor: Intel Corporation dmi.board.version: H57532-271 dmi.chassis.asset.tag: .................... dmi.chassis.type: 23 dmi.chassis.vendor: ............................... dmi.chassis.version: .................. dmi.modalias: dmi:bvnIntelCorporation:bvrS1200SP.86B.03.01.1029.012520180838:bd01/25/2018:svnIntelCorporation:pnS1200SP:pvr....................:rvnIntelCorporation:rnS1200SP:rvrH57532-271:cvn...............................:ct23:cvr..................: dmi.product.family: Family dmi.product.name: S1200SP dmi.product.version: .................... dmi.sys.vendor: Intel Corporation == SRU Justification == When using zram (as installed and configured with the zram-config package) systems can lockup after about a week of use. This occurs because of a hang in a lock in zram. == Test Case == Run stress-ng --brk 0 --stack 0 in a Bionic amd64 server VM with 1GM of memory, 16 CPU threads and zram-config installed. Without the fix the kernel will hang in a spinlock after 1-2 hours of run time. With the fix, the hang does not occur. Testing shows that with the fix, 5 x 16 CPU hours of stress testing with stress-ng works fine without the lockup occurring. == The fix == Upstream commit c4d6c4cc7bfd ("zram: correct flag name of ZRAM_ACCESS") as a prerequisite followed by a minor context wiggle backport of the fix with commit 3c9959e02547 ("zram: fix lockdep warning of free block handling"). == Regression Potential == This touches the zram locking, so the core zram driver is affected. However the fixes are backports from 5.0, so the fixes have had a fair amount of testing in later kernels. My main server has been running into hard lockups about once a week ever since I switched to the 4.15 Ubuntu 18.04 kernel. When this happens, nothing is printed to the console, it's effectively stuck showing a login prompt. The system is running with panic=1 on the cmdline but isn't rebooting so the kernel isn't even processing this as a kernel panic. As this felt like a potential hardware issue, I had my hosting provider give me a completely different system, different motherboard, different CPU, different RAM and different storage, I installed that system on 18.04 and moved my data over, a week later, I hit the issue again. We've since also had a LXD user reporting similar symptoms here also on varying hardware:   https://github.com/lxc/lxd/issues/5197 My system doesn't have a lot of memory pressure with about 50% of free memory: root@vorash:~# free -m               total used free shared buff/cache available Mem: 31819 17574 402 513 13842 13292 Swap: 15909 2687 13222 I will now try to increase console logging as much as possible on the system in the hopes that next time it hangs we can get a better idea of what happened but I'm not too hopeful given the complete silence on the console when this occurs. System is currently on:   Linux vorash 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux But I've seen this since the GA kernel on 4.15 so it's not a recent regression. --- ProblemType: Bug AlsaDevices:  total 0  crw-rw---- 1 root audio 116, 1 Oct 23 16:12 seq  crw-rw---- 1 root audio 116, 33 Oct 23 16:12 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay' ApportVersion: 2.20.9-0ubuntu7.4 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord' AudioDevicesInUse:  Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1: Cannot stat file /proc/22822/fd/10: Permission denied  Cannot stat file /proc/22831/fd/10: Permission denied DistroRelease: Ubuntu 18.04 HibernationDevice:  RESUME=none  CRYPTSETUP=n IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig' Lsusb:  Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub  Bus 001 Device 002: ID 046b:ff10 American Megatrends, Inc. Virtual Keyboard and Mouse  Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub MachineType: Intel Corporation S1200SP NonfreeKernelModules: zfs zunicode zavl icp zcommon znvpair Package: linux (not installed) PciMultimedia: ProcEnviron:  TERM=xterm  PATH=(custom, no user)  XDG_RUNTIME_DIR=<set>  LANG=en_US.UTF-8  SHELL=/bin/bash ProcFB: 0 mgadrmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-38-generic root=UUID=575c878a-0be6-4806-9c83-28f67aedea65 ro biosdevname=0 net.ifnames=0 panic=1 verbose console=tty0 console=ttyS0,115200n8 ProcVersionSignature: Ubuntu 4.15.0-38.41-generic 4.15.18 RelatedPackageVersions:  linux-restricted-modules-4.15.0-38-generic N/A  linux-backports-modules-4.15.0-38-generic N/A  linux-firmware 1.173.1 RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill' Tags: bionic Uname: Linux 4.15.0-38-generic x86_64 UnreportableReason: This report is about a package that is not installed. UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: _MarkForUpload: False dmi.bios.date: 01/25/2018 dmi.bios.vendor: Intel Corporation dmi.bios.version: S1200SP.86B.03.01.1029.012520180838 dmi.board.asset.tag: Base Board Asset Tag dmi.board.name: S1200SP dmi.board.vendor: Intel Corporation dmi.board.version: H57532-271 dmi.chassis.asset.tag: .................... dmi.chassis.type: 23 dmi.chassis.vendor: ............................... dmi.chassis.version: .................. dmi.modalias: dmi:bvnIntelCorporation:bvrS1200SP.86B.03.01.1029.012520180838:bd01/25/2018:svnIntelCorporation:pnS1200SP:pvr....................:rvnIntelCorporation:rnS1200SP:rvrH57532-271:cvn...............................:ct23:cvr..................: dmi.product.family: Family dmi.product.name: S1200SP dmi.product.version: .................... dmi.sys.vendor: Intel Corporation
2020-02-14 04:37:08 Khaled El Mously linux (Ubuntu Bionic): status Confirmed Fix Committed
2020-02-17 21:28:57 Ubuntu Kernel Bot tags apport-collected bionic cscc kernel-da-key apport-collected bionic cscc kernel-da-key verification-needed-bionic
2020-02-21 18:58:04 Colin Ian King tags apport-collected bionic cscc kernel-da-key verification-needed-bionic apport-collected bionic cscc kernel-da-key verification-done-bionic
2020-03-12 13:41:49 Colin Ian King bug task deleted zram-config (Ubuntu)
2020-03-12 13:42:00 Colin Ian King bug task deleted zram-config (Ubuntu Bionic)
2020-03-12 13:42:11 Colin Ian King linux (Ubuntu): status Incomplete Fix Committed
2020-03-16 10:53:57 Launchpad Janitor linux (Ubuntu Bionic): status Fix Committed Fix Released
2020-03-16 10:53:57 Launchpad Janitor cve linked 2019-14615
2020-03-16 10:53:57 Launchpad Janitor cve linked 2020-2732
2020-03-16 10:53:57 Launchpad Janitor cve linked 2020-8832
2020-03-16 19:11:46 Colin Ian King linux (Ubuntu): status Fix Committed Fix Released