kernel panic after upgrading from 19.04 to 19.10

[[e-mail]]

Upgrading from 19.04 to 19.10 on two of my pc does the same thing.
the upgrade is ok , boot ok, but after some time, and never the same, every app is crashing
unable to reboot.
i manage a reset with the pc button, but after that kernel panic and no boot.
every kernel listed is having a panic.
only chance after that is clonezilla and turn back to previous image 19.04 that i saved.
unable to enter recovery menu.
grub works
bios secure boot disabled.

   description: Computer
    width: 64 bits
    capabilities: smp vsyscall32
  *-core
       description: Motherboard
       physical id: 0
     *-memory
          description: System memory
          physical id: 0
          size: 13GiB
     *-cpu
          product: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz
          vendor: Intel Corp.
          physical id: 1
          bus info: cpu@0
          size: 3715MHz
          capacity: 3900MHz
          width: 64 bits
          capabilities: fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp x86-64 constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts md_clear flush_l1d cpufreq
     *-pci
          description: Host bridge
          product: Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller
          vendor: Intel Corporation
          physical id: 100
          bus info: pci@0000:00:00.0
          version: 09
          width: 32 bits
          clock: 33MHz
          configuration: driver=ivb_uncore
          resources: irq:0
        *-pci:0
             description: PCI bridge
             product: Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port
             vendor: Intel Corporation
             physical id: 1
             bus info: pci@0000:00:01.0
             version: 09
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:16 ioport:e000(size=4096) memory:f7e00000-f7efffff ioport:e0000000(size=268435456)
           *-display
                description: VGA compatible controller
                product: Turks XT [Radeon HD 6670/7670]
                vendor: Advanced Micro Devices, Inc. [AMD/ATI]
                physical id: 0
                bus info: pci@0000:01:00.0
                version: 00
                width: 64 bits
                clock: 33MHz
                capabilities: vga_controller bus_master cap_list rom
                configuration: driver=radeon latency=0
                resources: irq:27 memory:e0000000-efffffff memory:f7e20000-f7e3ffff ioport:e000(size=256) memory:c0000-dffff
           *-multimedia
                description: Audio device
                product: Turks HDMI Audio [Radeon HD 6500/6600 / 6700M Series]
                vendor: Advanced Micro Devices, Inc. [AMD/ATI]
                physical id: 0.1
                bus info: pci@0000:01:00.1
                version: 00
                width: 64 bits
                clock: 33MHz
                capabilities: bus_master cap_list
                configuration: driver=snd_hda_intel latency=0
                resources: irq:30 memory:f7e40000-f7e43fff
        *-usb:0
             description: USB controller
             product: 7 Series/C210 Series Chipset Family USB xHCI Host Controller
             vendor: Intel Corporation
             physical id: 14
             bus info: pci@0000:00:14.0
             version: 04
             width: 64 bits
             clock: 33MHz
             capabilities: xhci bus_master cap_list
             configuration: driver=xhci_hcd latency=0
             resources: irq:25 memory:f7f00000-f7f0ffff
        *-communication
             description: Communication controller
             product: 7 Series/C216 Chipset Family MEI Controller #1
             vendor: Intel Corporation
             physical id: 16
             bus info: pci@0000:00:16.0
             version: 04
             width: 64 bits
             clock: 33MHz
             capabilities: bus_master cap_list
             configuration: driver=mei_me latency=0
             resources: irq:28 memory:f7f1a000-f7f1a00f
        *-usb:1
             description: USB controller
             product: 7 Series/C216 Chipset Family USB Enhanced Host Controller #2
             vendor: Intel Corporation
             physical id: 1a
             bus info: pci@0000:00:1a.0
             version: 04
             width: 32 bits
             clock: 33MHz
             capabilities: ehci bus_master cap_list
             configuration: driver=ehci-pci latency=0
             resources: irq:16 memory:f7f18000-f7f183ff
        *-multimedia
             description: Audio device
             product: 7 Series/C216 Chipset Family High Definition Audio Controller
             vendor: Intel Corporation
             physical id: 1b
             bus info: pci@0000:00:1b.0
             version: 04
             width: 64 bits
             clock: 33MHz
             capabilities: bus_master cap_list
             configuration: driver=snd_hda_intel latency=0
             resources: irq:29 memory:f7f10000-f7f13fff
        *-pci:1
             description: PCI bridge
             product: 7 Series/C216 Chipset Family PCI Express Root Port 1
             vendor: Intel Corporation
             physical id: 1c
             bus info: pci@0000:00:1c.0
             version: c4
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:16
        *-pci:2
             description: PCI bridge
             product: 7 Series/C210 Series Chipset Family PCI Express Root Port 3
             vendor: Intel Corporation
             physical id: 1c.2
             bus info: pci@0000:00:1c.2
             version: c4
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:18 ioport:d000(size=4096) memory:f7d00000-f7dfffff
           *-network
                description: Ethernet interface
                product: AR8161 Gigabit Ethernet
                vendor: Qualcomm Atheros
                physical id: 0
                bus info: pci@0000:03:00.0
                logical name: enp3s0
                version: 10
                serial: 90:2b:34:5e:a2:2c
                size: 1Gbit/s
                capacity: 1Gbit/s
                width: 64 bits
                clock: 33MHz
                capabilities: bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
                configuration: autonegotiation=on broadcast=yes driver=alx duplex=full ip=192.168.0.2 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
                resources: irq:18 memory:f7d00000-f7d3ffff ioport:d000(size=128)
        *-pci:3
             description: PCI bridge
             product: 7 Series/C216 Chipset Family PCI Express Root Port 4
             vendor: Intel Corporation
             physical id: 1c.3
             bus info: pci@0000:00:1c.3
             version: c4
             width: 32 bits
             clock: 33MHz
             capabilities: pci normal_decode bus_master cap_list
             configuration: driver=pcieport
             resources: irq:19
           *-pci
                description: PCI bridge
                product: 82801 PCI Bridge
                vendor: Intel Corporation
                physical id: 0
                bus info: pci@0000:04:00.0
                version: 41
                width: 64 bits
                clock: 33MHz
                capabilities: pci subtractive_decode bus_master cap_list
                resources: iomemory:222001f10-222001f0f
        *-usb:2
             description: USB controller
             product: 7 Series/C216 Chipset Family USB Enhanced Host Controller #1
             vendor: Intel Corporation
             physical id: 1d
             bus info: pci@0000:00:1d.0
             version: 04
             width: 32 bits
             clock: 33MHz
             capabilities: ehci bus_master cap_list
             configuration: driver=ehci-pci latency=0
             resources: irq:23 memory:f7f17000-f7f173ff
        *-isa
             description: ISA bridge
             product: Z77 Express Chipset LPC Controller
             vendor: Intel Corporation
             physical id: 1f
             bus info: pci@0000:00:1f.0
             version: 04
             width: 32 bits
             clock: 33MHz
             capabilities: isa bus_master cap_list
             configuration: driver=lpc_ich latency=0
             resources: irq:0
        *-storage
             description: SATA controller
             product: 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
             vendor: Intel Corporation
             physical id: 1f.2
             bus info: pci@0000:00:1f.2
             version: 04
             width: 32 bits
             clock: 66MHz
             capabilities: storage ahci_1.0 bus_master cap_list
             configuration: driver=ahci latency=0
             resources: irq:26 ioport:f070(size=8) ioport:f060(size=4) ioport:f050(size=8) ioport:f040(size=4) ioport:f020(size=32) memory:f7f16000-f7f167ff
        *-serial UNCLAIMED
             description: SMBus
             product: 7 Series/C216 Chipset Family SMBus Controller
             vendor: Intel Corporation
             physical id: 1f.3
             bus info: pci@0000:00:1f.3
             version: 04
             width: 64 bits
             clock: 33MHz
             configuration: latency=0
             resources: memory:f7f15000-f7f150ff ioport:f000(size=32)

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1849419

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[[e-mail] (r-fabbeni)]

Impossible to run such command because every command crashes after kernel panic
And no chance to get in recovery menu

[[e-mail] (r-fabbeni)]

Also happens in my other laptop Asus k53sv.
After some minutes or some hour commands start crashing
And it's impossible to execute any command.
Apport is not opening.
Shutdown is impossible

[[e-mail] (r-fabbeni)]

Oct 27 03:29:25 firefox-server kernel: [ 6.575688] alx 0000:03:00.0 enp3s0: NIC Up: 1 Gbps Full
Oct 27 03:29:25 firefox-server kernel: [ 6.575896] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready
Oct 27 03:29:28 firefox-server kernel: [ 9.157343] radeon_dp_aux_transfer_native: 158 callbacks suppressed
Oct 27 03:29:30 firefox-server kernel: [ 11.005322] usb 1-1.5.3: reset high-speed USB device number 8 using ehci-pci
Oct 27 03:29:32 firefox-server kernel: [ 12.696160] Bluetooth: RFCOMM TTY layer initialized
Oct 27 03:29:32 firefox-server kernel: [ 12.696166] Bluetooth: RFCOMM socket layer initialized
Oct 27 03:29:32 firefox-server kernel: [ 12.696170] Bluetooth: RFCOMM ver 1.11
Oct 27 03:29:34 firefox-server kernel: [ 14.853182] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)
Oct 27 04:30:33 firefox-server kernel: [ 74.266876] radeon_dp_aux_transfer_native: 326 callbacks suppressed
Oct 27 04:31:12 firefox-server kernel: [ 113.243986] kauditd_printk_skb: 27 callbacks suppressed
Oct 27 04:31:12 firefox-server kernel: [ 113.243987] audit: type=1400 audit(1572147072.420:38): apparmor="DENIED" operation="open" profile="/usr/bin/msmtp" name="/home/renato/.msmtp" pid=15180 comm="msmtp" requested_mask="ac" denied_mask="ac" fsuid=1000 ouid=1000
Oct 27 04:34:43 firefox-server kernel: [ 324.181111] audit: type=1400 audit(1572147283.354:39): apparmor="DENIED" operation="open" profile="/usr/bin/msmtp" name="/home/renato/.msmtp" pid=2989 comm="msmtp" requested_mask="ac" denied_mask="ac" fsuid=1000 ouid=1000
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] microcode: microcode updated early to revision 0x21, date = 2019-02-13
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] Linux version 5.0.0-32-generic (buildd@lgw01-amd64-034) (gcc version 8.3.0 (Ubuntu 8.3.0-6ubuntu1)) #34-Ubuntu SMP Wed Oct 2 02:06:48 UTC 2019 (Ubuntu 5.0.0-32.34-generic 5.0.21)
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.0.0-32-generic root=UUID=acdb3f52-4b69-48c4-8fda-73500fcb8a33 ro quiet splash vt.handoff=1
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] KERNEL supported cpus:
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] Intel GenuineIntel
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] AMD AuthenticAMD
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] Hygon HygonGenuine
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] Centaur CentaurHauls
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
Oct 27 04:58:22 firefox-server kernel: [ 0.000000] BIOS-provided physical RAM map:
Oct 27 04:58:22 f...

[[e-mail] (r-fabbeni)]

I did upgrade again.
19.04 to 19.10

after poweroff for all night. the morning i switch pc on and after a few minutes i get the same kernel panic on every kernel installed.
i did test kernel 5.4 rc3 and no success.
same bug
i am starting to think it's related to deluge because i was adding a magnet torrent and everything crashed.
every application command had i sigfault.

[[e-mail] (r-fabbeni)]

please help, it's a week i am testing everything with no success

[[e-mail] (r-fabbeni)]

After many testing i discoverd that was depending on a bash script where the command
shutdown +3 was crashing everything
i changed to shutdown now and the problem seems solved

[[e-mail] (r-fabbeni)]

That solved for one day.
After a few minutes crashed again.
And it is not a crib script
Reverting to 19.04 works again

[You-Sheng Yang (vicamo)]

From dmesg in comment #5:

kernel: [ 0.000000] Linux version 5.3.0-19-generic (buildd@lgw01-amd64-053) (gcc version 9.2.1 20191008 (Ubuntu 9.2.1-9ubuntu2)) #20-Ubuntu SMP Fri Oct 18 09:04:39 UTC 2019 (Ubuntu 5.3.0-19.20-generic 5.3.1)
...
kernel: [ 327.012856] cut[7266]: segfault at 0 ip 00007fe79323a1f6 sp 00007fffeb8c14f8 error 6 in libc-2.30.so[7fe79323a000+178000]
kernel: [ 327.012862] Code: fe ff ff 90 f3 0f 1e fa 68 19 00 00 00 f2 e9 21 fe ff ff 90 f3 0f 1e fa 68 18 00 00 00 f2 e9 11 fe ff ff 90 f3 0f 1e fa 68 05 <00> 00 00 f2 e9 01 fe ff ff 90 f3 0f 1e fa 68 17 00 00 00 f2 e9 f1
kernel: [ 327.012928] sed[7265]: segfault at 0 ip 00007f2c0794b506 sp 00007ffe7da24728 error 6 in libpthread-2.30.so[7f2c0794b000+11000]
kernel: [ 327.012933] Code: fb ff ff 90 f3 0f 1e fa 68 4b 00 00 00 f2 e9 31 fb ff ff 90 f3 0f 1e fa 68 4c 00 00 00 f2 e9 21 fb ff ff 90 f3 0f 1e fa 68 4d <00> 00 00 f2 e9 11 fb ff ff 90 f3 0f 1e fa 68 4e 00 00 00 f2 e9 01
kernel: [ 327.012944] ifstat[7263]: segfault at 0 ip 00007f7e3db93506 sp 00007ffd4cacef28 error 6 in libpthread-2.30.so[7f7e3db93000+11000]
kernel: [ 327.012948] Code: fb ff ff 90 f3 0f 1e fa 68 4b 00 00 00 f2 e9 31 fb ff ff 90 f3 0f 1e fa 68 4c 00 00 00 f2 e9 21 fb ff ff 90 f3 0f 1e fa 68 4d <00> 00 00 f2 e9 11 fb ff ff 90 f3 0f 1e fa 68 4e 00 00 00 f2 e9 01
....

The kernel boots fine. At first glance it seems to me somehow you get either corrupted filesystem or corrupted binaries installed. With some more googling, it could also be bad memory module installed[1]. Could you try removing/replacing them?

[1]: https://askubuntu.com/questions/975828/how-can-i-find-the-cause-of-the-segfaults-im-receiving-from-a-fresh-17-10-insta?rq=1

[[e-mail] (r-fabbeni)]

Will test it tonight thanks

[[e-mail] (r-fabbeni)]

But bad memory I should get errors even on 19.04 and there isnt anything .
But will try

[[e-mail] (r-fabbeni)]

Did a memory test .
Passed.
And after I realized that could not be a hardware issue because I have the same problem on my laptop too.

[[e-mail] (r-fabbeni)]

i pulled out all ram and cleaned the contacts.
pulled out all connectors and replaced them in/out a few times.
upgraded to 19.10 now.
I did it via ssh because a few times i got black screen in the middle of upgrade and a half way upgrade that i had to finish up booting in rescue mode.
will test it a day or two and let you know.
even if it passed all ram test and disk test , but really strange. did the same thing on my laptop the same day and i doubt that even that one needs to be cleaned too.

[[e-mail] (r-fabbeni)]

Solved dirty ram was hardware prob

[[e-mail] (r-fabbeni)]

one day and again kernel crash.
removed every ram from the pc and set up new ones.
nothing to do.
other pc same story

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1849419

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[[e-mail] (r-fabbeni)]

Upgraded again, and same result. tested filesystem, ram everything. system runs fine
and after i switch off the pc for some hours will not start again.

[Chris Guiver (guiverc)]

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 1849419

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[[e-mail] (r-fabbeni)]

ok done

[[e-mail] (r-fabbeni)]

Upgraded again this morning. Disabled everything in cron .
The PC was all day running . At midnight about at 00.45 I had a terminal opend and I executed a program... Crashed.. every program is crashing.. no one working. Segmentation faults everywhere .
Rebooted because noway to do anything and every kernel was not booting up. I had to restore again the 19.04 with clonezilla.. this is driving me crazy.
I saved the var log dir but not as root . Sudo was crashing as all other .. no way ..
What should I do now ?

[You-Sheng Yang (vicamo)]

I would recommend you to move the whole disk to another known working machine when this happens again to make sure this is not a hardware issue.

As for /var directory you saved, root is root universally. You should be able to examine its content in some live CD session or so.

[[e-mail] (r-fabbeni)]

ERROR: apport (pid 5293) Wed Dec 4 00:32:30 2019: called for pid 4629, signal 6, core limit 0, dump mode 1
ERROR: apport (pid 5293) Wed Dec 4 00:32:30 2019: executable: /usr/lib/tracker/tracker-miner-fs (command line "/usr/lib/tracker/tracker-miner-fs")
ERROR: apport (pid 5293) Wed Dec 4 00:32:30 2019: debug: session gdbus call: (true,)

ERROR: apport (pid 5293) Wed Dec 4 00:32:30 2019: this executable already crashed 2 times, ignoring

i found this in apport.log after this everything seems crashing. i cant move the entire disck sorry it's a server and would cause noo many problems

[[e-mail] (r-fabbeni)]

it cant be an hardware issue because with 19.04 works fine all day forever. after 19.10 crash.
i did a phoronix test and was all ok too.

[[e-mail] (r-fabbeni)]

new test after googling.
removed preload and prelinc before upgrading.
upgraded and after midnight at 1.00 am i switched off. was all ok.
maybe a bug with those two programs ?

[[e-mail] (r-fabbeni)]

Removed preload and prelink from the two pc .
Upgraded to 19.10.
Two days running. A ok now.
Preload and prelink cause the kernel panic .
Please take care about this bug bacause will drive crazy everyone that
Has these installed .
Maybe we have to open a bugreport specific to sign out this prob ?

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]