This issue was found after leaving a SUT to run overnight (was testing the ubuntu_cts_kernel test 13 hours ago before this happens). Can't tell if this is a regression, as I haven't find a way to reproduce it. But I do see similar reports on the Internet [1]
After checking the systemd service timers, it looks like this is caused by the logrotate.service:
$ sudo systemctl list-timers --all
NEXT LEFT LAST PASSED UNIT ACTIVATES
Fri 2019-01-25 06:18:58 UTC 1h 40min left Thu 2019-01-24 06:34:15 UTC 22h ago apt-daily-upgrade.timer apt-daily-upgrade.service
Fri 2019-01-25 09:15:54 UTC 4h 37min left Fri 2019-01-25 03:43:24 UTC 55min ago apt-daily.timer apt-daily.service
Fri 2019-01-25 17:02:47 UTC 12h left Fri 2019-01-25 04:20:17 UTC 18min ago motd-news.timer motd-news.service
Sat 2019-01-26 00:00:00 UTC 19h left Fri 2019-01-25 00:00:37 UTC 4h 38min ago logrotate.timer logrotate.service
Sat 2019-01-26 04:02:38 UTC 23h left Fri 2019-01-25 04:02:38 UTC 36min ago systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.serv
Mon 2019-01-28 00:00:00 UTC 2 days left Wed 2019-01-23 10:44:18 UTC 1 day 17h ago fstrim.timer fstrim.service
n/a n/a n/a n/a snapd.snap-repair.timer snapd.snap-repair.service
7 timers listed.
After this happens, you won't be able to reboot it with the command.
$ sudo reboot
sudo: unable to resolve host onza: Resource temporarily unavailable
Killed
Here is the error message, please refer to the attachment for a complete syslog:
ProblemType: Bug
DistroRelease: Ubuntu 18.10
Package: linux-image-4.18.0-14-generic 4.18.0-14.15
ProcVersionSignature: User Name 4.18.0-14.15-generic 4.18.20
Uname: Linux 4.18.0-14-generic i686
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jan 25 03:48 seq
crw-rw---- 1 root audio 116, 33 Jan 25 03:48 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.10-0ubuntu13.1
Architecture: i386
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDmesg:
Date: Fri Jan 25 04:18:08 2019
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Dell Inc. PowerEdge R310
PciMultimedia:
ProcFB: 0 mgadrmfb
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.18.0-14-generic root=UUID=6aaa11f6-d386-4c0c-b4b8-38e6c408980a ro console=ttyS0,115200n8
RelatedPackageVersions:
linux-restricted-modules-4.18.0-14-generic N/A
linux-backports-modules-4.18.0-14-generic N/A
linux-firmware 1.175.1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 09/18/2012
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.11.0
dmi.board.name: 05XKKK
dmi.board.vendor: Dell Inc.
dmi.board.version: A05
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvr1.11.0:bd09/18/2012:svnDellInc.:pnPowerEdgeR310:pvr:rvnDellInc.:rn05XKKK:rvrA05:cvnDellInc.:ct23:cvr:
dmi.product.name: PowerEdge R310
dmi.sys.vendor: Dell Inc.
This issue was found after leaving a SUT to run overnight (was testing the ubuntu_cts_kernel test 13 hours ago before this happens). Can't tell if this is a regression, as I haven't find a way to reproduce it. But I do see similar reports on the Internet [1]
After checking the systemd service timers, it looks like this is caused by the logrotate.service:
$ sudo systemctl list-timers --all upgrade. timer apt-daily- upgrade. service tmpfiles- clean.timer systemd- tmpfiles- clean.serv repair. timer snapd.snap- repair. service
NEXT LEFT LAST PASSED UNIT ACTIVATES
Fri 2019-01-25 06:18:58 UTC 1h 40min left Thu 2019-01-24 06:34:15 UTC 22h ago apt-daily-
Fri 2019-01-25 09:15:54 UTC 4h 37min left Fri 2019-01-25 03:43:24 UTC 55min ago apt-daily.timer apt-daily.service
Fri 2019-01-25 17:02:47 UTC 12h left Fri 2019-01-25 04:20:17 UTC 18min ago motd-news.timer motd-news.service
Sat 2019-01-26 00:00:00 UTC 19h left Fri 2019-01-25 00:00:37 UTC 4h 38min ago logrotate.timer logrotate.service
Sat 2019-01-26 04:02:38 UTC 23h left Fri 2019-01-25 04:02:38 UTC 36min ago systemd-
Mon 2019-01-28 00:00:00 UTC 2 days left Wed 2019-01-23 10:44:18 UTC 1 day 17h ago fstrim.timer fstrim.service
n/a n/a n/a n/a snapd.snap-
7 timers listed.
After this happens, you won't be able to reboot it with the command.
$ sudo reboot
sudo: unable to resolve host onza: Resource temporarily unavailable
Killed
Here is the error message, please refer to the attachment for a complete syslog:
Jan 25 00:00:37 onza systemd[1]: Starting Rotate log files... 0xc9/0x240 store_dirent+ 0x2e/0x120 store_dirent+ 0x2e/0x120 to_tree+ 0xd2/0x230 fill_tree+ 0x7f/0x2c0 alloc_trace+ 0x167/0x1e0 0x9d4/0xb10 0x6aa/0xb10 user+0x21/ 0x30 0x34/0x60 file_permission +0x7c/0xb0 dir+0x15c/ 0x1a0 +0x6f/0x110 dir+0x1a0/ 0x1a0 0x16/0x20 syscall_ 32+0x7f/ 0x1e0 32+0x4e/ 0x7c iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mgag200 i2c_algo_bit ttm hid_generic drm_kms_helper gpio_ich syscopyarea sysfillrect sysimgblt mpt3sas fb_sys_fops drm
Jan 25 00:00:37 onza kernel: [45282.956634] BUG: unable to handle kernel paging request at 6db23a14
Jan 25 00:00:37 onza kernel: [45282.962902] *pdpt = 0000000033fa1001 *pde = 0000000000000000
Jan 25 00:00:37 onza kernel: [45282.968650] Oops: 0000 [#1] SMP
Jan 25 00:00:37 onza kernel: [45282.971796] CPU: 0 PID: 407 Comm: systemd-journal Not tainted 4.18.0-14-generic #15-Ubuntu
Jan 25 00:00:37 onza kernel: [45282.980330] Hardware name: Dell Inc. PowerEdge R310/05XKKK, BIOS 1.11.0 09/18/2012
Jan 25 00:00:37 onza kernel: [45282.988127] EIP: __kmalloc+
Jan 25 00:00:37 onza kernel: [45282.991785] Code: 4d e0 8d 4a 01 31 c6 8b 45 f0 89 75 dc 31 f3 8b 37 64 0f c7 0e 0f 94 c0 84 c0 74 b9 8b 75 e4 8b 45 dc 39 45 e0 74 0e 03 5f 14 <33> 1b 33 9f b4 00 00 00 0f 18 03 f7 45 ec 00 80 00 00 0f 85 3f 01
Jan 25 00:00:37 onza kernel: [45283.011254] EAX: 9e20b374 EBX: 6db23a14 ECX: 0000350d EDX: 0000350c
Jan 25 00:00:37 onza kernel: [45283.017769] ESI: eac03a00 EDI: eac03a00 EBP: f461fd9c ESP: f461fd74
Jan 25 00:00:37 onza kernel: [45283.024304] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010206
Jan 25 00:00:37 onza kernel: [45283.031245] CR0: 80050033 CR2: 6db23a14 CR3: 29d304a0 CR4: 000006f0
Jan 25 00:00:37 onza kernel: [45283.037770] Call Trace:
Jan 25 00:00:37 onza kernel: [45283.040395] ? ext4_htree_
Jan 25 00:00:37 onza kernel: [45283.045255] ext4_htree_
Jan 25 00:00:37 onza kernel: [45283.049951] htree_dirblock_
Jan 25 00:00:37 onza kernel: [45283.054562] ext4_htree_
Jan 25 00:00:37 onza kernel: [45283.058991] ? kmem_cache_
Jan 25 00:00:37 onza kernel: [45283.063849] ? ext4_readdir+
Jan 25 00:00:37 onza kernel: [45283.067854] ext4_readdir+
Jan 25 00:00:37 onza kernel: [45283.071687] ? _copy_to_
Jan 25 00:00:37 onza kernel: [45283.075597] ? do_statfs64+
Jan 25 00:00:37 onza kernel: [45283.079334] ? security_
Jan 25 00:00:37 onza kernel: [45283.084306] iterate_
Jan 25 00:00:37 onza kernel: [45283.087883] ksys_getdents64
Jan 25 00:00:37 onza kernel: [45283.091853] ? iterate_
Jan 25 00:00:37 onza kernel: [45283.095771] sys_getdents64+
Jan 25 00:00:37 onza kernel: [45283.099595] do_fast_
Jan 25 00:00:37 onza kernel: [45283.103858] entry_SYSENTER_
Jan 25 00:00:37 onza kernel: [45283.107938] EIP: 0xb7ef4d41
Jan 25 00:00:37 onza kernel: [45283.110898] Code: f6 ff ff 55 89 e5 8b 55 08 8b 80 5c cd ff ff 85 d2 74 02 89 02 5d c3 8b 04 24 c3 8b 1c 24 c3 90 90 51 52 55 89 e5 0f 34 cd 80 <5d> 5a 59 c3 90 90 90 90 8d 76 00 58 b8 77 00 00 00 cd 80 90 8d 76
Jan 25 00:00:37 onza kernel: [45283.130298] EAX: ffffffda EBX: 00000021 ECX: 02219dec EDX: 00008000
Jan 25 00:00:37 onza kernel: [45283.136824] ESI: ffffffb4 EDI: 00000000 EBP: 02219dd0 ESP: bff8f5e8
Jan 25 00:00:37 onza kernel: [45283.143245] DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 007b EFLAGS: 00000296
Jan 25 00:00:37 onza kernel: [45283.150299] Modules linked in: iptable_nat ip_gre gre ip_tunnel dummy iptable_filter xt_recent bpfilter openvswitch nsh nf_conntrack_ipv6 nf_nat_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_defrag_ipv6 nf_conncount nf_nat nf_conntrack intel_powerclamp coretemp ipmi_ssif kvm_intel kvm dcdbas ipmi_si joydev irqbypass intel_cstate input_leds ipmi_devintf i7core_edac ipmi_msghandler acpi_power_meter mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_
Jan 25 00:00:37 onza kernel: [45283.222723] usbhid raid_class pata_acpi lpc_ich hid scsi_transport_sas bnx2 wmi
Jan 25 00:00:37 onza kernel: [45283.230378] CR2: 000000006db23a14
Jan 25 00:00:37 onza kernel: [45283.233894] ---[ end trace a12425d45f6ac772 ]---
[1] https:/ /forum. manjaro. org/t/systemd- timer-crashing- my-system- at-midnight/ 19609
[2]
ProblemType: Bug 4.18.0- 14-generic 4.18.0-14.15 ature: User Name 4.18.0- 14.15-generic 4.18.20
DistroRelease: Ubuntu 18.10
Package: linux-image-
ProcVersionSign
Uname: Linux 4.18.0-14-generic i686
AlsaDevices:
total 0
crw-rw---- 1 root audio 116, 1 Jan 25 03:48 seq
crw-rw---- 1 root audio 116, 33 Jan 25 03:48 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay': 'aplay'
ApportVersion: 2.20.10-0ubuntu13.1
Architecture: i386
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord': 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CurrentDmesg:
Date: Fri Jan 25 04:18:08 2019
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig': 'iwconfig'
MachineType: Dell Inc. PowerEdge R310
PciMultimedia:
ProcFB: 0 mgadrmfb /boot/vmlinuz- 4.18.0- 14-generic root=UUID= 6aaa11f6- d386-4c0c- b4b8-38e6c40898 0a ro console= ttyS0,115200n8 ersions: restricted- modules- 4.18.0- 14-generic N/A backports- modules- 4.18.0- 14-generic N/A :bvr1.11. 0:bd09/ 18/2012: svnDellInc. :pnPowerEdgeR31 0:pvr:rvnDellIn c.:rn05XKKK: rvrA05: cvnDellInc. :ct23:cvr:
ProcKernelCmdLine: BOOT_IMAGE=
RelatedPackageV
linux-
linux-
linux-firmware 1.175.1
RfKill: Error: [Errno 2] No such file or directory: 'rfkill': 'rfkill'
SourcePackage: linux
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 09/18/2012
dmi.bios.vendor: Dell Inc.
dmi.bios.version: 1.11.0
dmi.board.name: 05XKKK
dmi.board.vendor: Dell Inc.
dmi.board.version: A05
dmi.chassis.type: 23
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.
dmi.product.name: PowerEdge R310
dmi.sys.vendor: Dell Inc.