Kernel panic - not syncing: IO-APIC + timer doesn't work!

[Lee Yarwood]

https://zuul.opendev.org/t/openstack/build/547649e867374c04bad3b29a82cb3132/log/compute1/logs/screen-n-cpu.txt#14687

Oct 07 12:07:03.482952 ubuntu-focal-ovh-bhs1-0026837355 nova-compute[68013]: <os>
Oct 07 12:07:03.482952 ubuntu-focal-ovh-bhs1-0026837355 nova-compute[68013]: <type machine="pc">hvm</type>
Oct 07 12:07:03.482952 ubuntu-focal-ovh-bhs1-0026837355 nova-compute[68013]: <boot dev="hd"/>
Oct 07 12:07:03.482952 ubuntu-focal-ovh-bhs1-0026837355 nova-compute[68013]: <smbios mode="sysinfo"/>
Oct 07 12:07:03.482952 ubuntu-focal-ovh-bhs1-0026837355 nova-compute[68013]: </os>

https://4466335acde89930c709-42b9b3ca9891e58d539431fcfb5b799d.ssl.cf5.rackcdn.com/812392/3/check/nova-multi-cell/547649e/testr_results.html

2021-10-07 12:14:37,051 120294 INFO [tempest.lib.common.rest_client] Request (ServersTestManualDisk:test_host_name_is_same_as_server_name): 200 POST https://158.69.68.56/compute/v2.1/servers/3a336643-47eb-4231-a1a6-b9bc62f88f95/action 0.938s
2021-10-07 12:14:37,052 120294 DEBUG [tempest.lib.common.rest_client] Request - Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<omitted>'}
        Body: {"os-getConsoleOutput": {}}
    Response - Headers: {'date': 'Thu, 07 Oct 2021 12:14:36 GMT', 'server': 'Apache/2.4.41 (Ubuntu)', 'content-length': '7775', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.1', 'x-openstack-nova-api-version': '2.1', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-bf5504f3-c8bf-4163-8eca-f2bf126eb60a', 'x-compute-request-id': 'req-bf5504f3-c8bf-4163-8eca-f2bf126eb60a', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.68.56/compute/v2.1/servers/3a336643-47eb-4231-a1a6-b9bc62f88f95/action'}
        Body: b'{"output": "[ 0.000000] Linux version 5.3.0-26-generic (buildd@lgw01-amd64-039) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 (Ubuntu 5.3.0-26.28~18.04.1-generic 5.3.13)\\n[ 0.000000] Command line: LABEL=cirros-rootfs ro console=tty1 console=ttyS0\\n[ 0.000000] KERNEL supported cpus:\\n[ 0.000000] Intel GenuineIntel\\n[ 0.000000] AMD AuthenticAMD\\n[ 0.000000] Hygon HygonGenuine\\n[ 0.000000] Centaur CentaurHauls\\n[ 0.000000] zhaoxin Shanghai \\n[ 0.000000] x86/fpu: x87 FPU will use FXSAVE\\n[ 0.000000] BIOS-provided physical RAM map:\\n[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable\\n[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved\\n[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved\\n[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000007fdcfff] usable\\n[ 0.000000] BIOS-e820: [mem 0x0000000007fdd000-0x0000000007ffffff] reserved\\n[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved\\n[ 0.000000] NX (Execute Disable) protection: active\\n[ 0.000000] SMBIOS 2.8 present.\\n[ 0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 1.13.0-1ubuntu1.1 04/01/2014\\n[ 0.000000] tsc: Fast TSC calibration using PIT\\n[ 0.000000] tsc: Detected 2399.991 MHz processor\\n[ 0.024611] last_pfn = 0x7fdd max_arch_pfn = 0x400000000\\n[ 0.026398] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT \\n[ 0.055021] found SMP MP-table at [mem 0x000f5c80-0x000f5c8f]\\n[ 0.074763] check: Scanning 1 areas for low memory corruption\\n[ 0.082155] RAMDISK: [mem 0x0798e000-0x07fccfff]\\n[ 0.082934] ACPI: Early table checksum verification disabled\\n[ 0.083702] ACPI: RSDP 0x00000000000F5A90 000014 (v00 BOCHS )\\n[ 0.084124] ACPI: RSDT 0x0000000007FE153C 000030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)\\n[ 0.085156] ACPI: FACP 0x0000000007FE1418 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)\\n[ 0.086081] ACPI: DSDT 0x0000000007FE0040 0013D8 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)\\n[ 0.086212] ACPI: FACS 0x0000000007FE0000 000040\\n[ 0.086319] ACPI: APIC 0x0000000007FE148C 000078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)\\n[ 0.086372] ACPI: HPET 0x0000000007FE1504 000038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001)\\n[ 0.089699] No NUMA configuration found\\n[ 0.089736] Faking a node at [mem 0x0000000000000000-0x0000000007fdcfff]\\n[ 0.090618] NODE_DATA(0) allocated [mem 0x07963000-0x0798dfff]\\n[ 0.095595] Zone ranges:\\n[ 0.095707] DMA [mem 0x0000000000001000-0x0000000000ffffff]\\n[ 0.095766] DMA32 [mem 0x0000000001000000-0x0000000007fdcfff]\\n[ 0.095783] Normal empty\\n[ 0.095808] Device empty\\n[ 0.095827] Movable zone start for each node\\n[ 0.095914] Early memory node ranges\\n[ 0.095965] node 0: [mem 0x0000000000001000-0x000000000009efff]\\n[ 0.096197] node 0: [mem 0x0000000000100000-0x0000000007fdcfff]\\n[ 0.096542] Zeroed struct page in unavailable ranges: 98 pages\\n[ 0.096762] Initmem setup node 0 [mem 0x0000000000001000-0x0000000007fdcfff]\\n[ 0.102504] ACPI: PM-Timer IO Port: 0x608\\n[ 0.103095] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])\\n[ 0.103612] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23\\n[ 0.103778] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)\\n[ 0.104191] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)\\n[ 0.104242] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)\\n[ 0.104362] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)\\n[ 0.104380] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)\\n[ 0.104655] Using ACPI (MADT) for SMP configuration information\\n[ 0.104724] ACPI: HPET id: 0x8086a201 base: 0xfed00000\\n[ 0.105183] smpboot: Allowing 1 CPUs, 0 hotplug CPUs\\n[ 0.106164] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]\
2021-10-07 12:14:37,053 120294 DEBUG [tempest.lib.common.utils.linux.remote_client] Console log for server 3a336643-47eb-4231-a1a6-b9bc62f88f95: [ 0.000000] Linux version 5.3.0-26-generic (buildd@lgw01-amd64-039) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04.1)) #28~18.04.1-Ubuntu SMP Wed Dec 18 16:40:14 UTC 2019 (Ubuntu 5.3.0-26.28~18.04.1-generic 5.3.13)
[ 0.000000] Command line: LABEL=cirros-rootfs ro console=tty1 console=ttyS0
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Hygon HygonGenuine
[ 0.000000] Centaur CentaurHauls
[ 0.000000] zhaoxin Shanghai
[ 0.000000] x86/fpu: x87 FPU will use FXSAVE
[ 0.000000] BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000007fdcfff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000007fdd000-0x0000000007ffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] SMBIOS 2.8 present.
[ 0.000000] DMI: OpenStack Foundation OpenStack Nova, BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 0.000000] tsc: Fast TSC calibration using PIT
[ 0.000000] tsc: Detected 2399.991 MHz processor
[ 0.024611] last_pfn = 0x7fdd max_arch_pfn = 0x400000000
[ 0.026398] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT
[ 0.055021] found SMP MP-table at [mem 0x000f5c80-0x000f5c8f]
[ 0.074763] check: Scanning 1 areas for low memory corruption
[ 0.082155] RAMDISK: [mem 0x0798e000-0x07fccfff]
[ 0.082934] ACPI: Early table checksum verification disabled
[ 0.083702] ACPI: RSDP 0x00000000000F5A90 000014 (v00 BOCHS )
[ 0.084124] ACPI: RSDT 0x0000000007FE153C 000030 (v01 BOCHS BXPCRSDT 00000001 BXPC 00000001)
[ 0.085156] ACPI: FACP 0x0000000007FE1418 000074 (v01 BOCHS BXPCFACP 00000001 BXPC 00000001)
[ 0.086081] ACPI: DSDT 0x0000000007FE0040 0013D8 (v01 BOCHS BXPCDSDT 00000001 BXPC 00000001)
[ 0.086212] ACPI: FACS 0x0000000007FE0000 000040
[ 0.086319] ACPI: APIC 0x0000000007FE148C 000078 (v01 BOCHS BXPCAPIC 00000001 BXPC 00000001)
[ 0.086372] ACPI: HPET 0x0000000007FE1504 000038 (v01 BOCHS BXPCHPET 00000001 BXPC 00000001)
[ 0.089699] No NUMA configuration found
[ 0.089736] Faking a node at [mem 0x0000000000000000-0x0000000007fdcfff]
[ 0.090618] NODE_DATA(0) allocated [mem 0x07963000-0x0798dfff]
[ 0.095595] Zone ranges:
[ 0.095707] DMA [mem 0x0000000000001000-0x0000000000ffffff]
[ 0.095766] DMA32 [mem 0x0000000001000000-0x0000000007fdcfff]
[ 0.095783] Normal empty
[ 0.095808] Device empty
[ 0.095827] Movable zone start for each node
[ 0.095914] Early memory node ranges
[ 0.095965] node 0: [mem 0x0000000000001000-0x000000000009efff]
[ 0.096197] node 0: [mem 0x0000000000100000-0x0000000007fdcfff]
[ 0.096542] Zeroed struct page in unavailable ranges: 98 pages
[ 0.096762] Initmem setup node 0 [mem 0x0000000000001000-0x0000000007fdcfff]
[ 0.102504] ACPI: PM-Timer IO Port: 0x608
[ 0.103095] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[ 0.103612] IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
[ 0.103778] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[ 0.104191] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[ 0.104242] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[ 0.104362] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[ 0.104380] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[ 0.104655] Using ACPI (MADT) for SMP configuration information
[ 0.104724] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[ 0.105183] smpboot: Allowing 1 CPUs, 0 hotplug CPUs
[ 0.106164] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[ 0.106231] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[ 0.106264] PM: Registered nosave memory: [mem 0x000a0000-0x000effff]
[ 0.106276] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[ 0.106413] [mem 0x08000000-0xfffbffff] available for PCI devices
[ 0.106465] Booting paravirtualized kernel on bare hardware
[ 0.106955] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[ 0.107487] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
[ 0.117766] percpu: Embedded 54 pages/cpu s184320 r8192 d28672 u2097152
[ 0.120755] Built 1 zonelists, mobility grouping on. Total pages: 32102
[ 0.120813] Policy zone: DMA32
[ 0.121072] Kernel command line: LABEL=cirros-rootfs ro console=tty1 console=ttyS0
[ 0.123070] Dentry cache hash table entries: 16384 (order: 5, 131072 bytes, linear)
[ 0.123434] Inode-cache hash table entries: 8192 (order: 4, 65536 bytes, linear)
[ 0.125940] mem auto-init: stack:off, heap alloc:on, heap free:off
[ 0.132037] Memory: 88564K/130540K available (14339K kernel code, 2370K rwdata, 4668K rodata, 2660K init, 5076K bss, 41976K reserved, 0K cma-reserved)
[ 0.133043] random: get_random_u64 called from __kmem_cache_create+0x41/0x540 with crng_init=0
[ 0.142582] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[ 0.144696] ftrace: allocating 42927 entries in 168 pages
[ 0.280917] rcu: Hierarchical RCU implementation.
[ 0.281077] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=1.
[ 0.281188] Tasks RCU enabled.
[ 0.281272] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[ 0.281317] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=1
[ 0.314534] NR_IRQS: 524544, nr_irqs: 256, preallocated irqs: 16
[ 0.331474] Console: colour VGA+ 80x25
[ 0.336508] printk: console [tty1] enabled
[ 0.388671] printk: console [ttyS0] enabled
[ 0.391139] ACPI: Core revision 20190703
[ 0.399479] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns
[ 0.407394] APIC: Switch to symmetric I/O mode setup
[ 0.415185] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1
[ 0.483446] ..MP-BIOS bug: 8254 timer not connected to IO-APIC
[ 0.485064] ...trying to set up timer (IRQ0) through the 8259A ...
[ 0.486568] ..... (found apic 0 pin 2) ...
[ 0.554482] ....... failed.
[ 0.555301] ...trying to set up timer as Virtual Wire IRQ...
[ 0.624032] ..... failed.
[ 0.625124] ...trying to set up timer as ExtINT IRQ...
[ 2.730994] ..... failed :(.
[ 2.731567] Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option.
[ 2.732385] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.3.0-26-generic #28~18.04.1-Ubuntu
[ 2.732733] Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.13.0-1ubuntu1.1 04/01/2014
[ 2.733173] Call Trace:
[ 2.734380] dump_stack+0x6d/0x95
[ 2.734605] panic+0xfe/0x2d4
[ 2.734767] check_timer+0x5e8/0x685
[ 2.734938] ? radix_tree_lookup+0xd/0x10
[ 2.735136] setup_IO_APIC+0x182/0x1ca
[ 2.735275] apic_intr_mode_init+0x1f5/0x1f8
[ 2.735419] x86_late_time_init+0x1b/0x22
[ 2.735595] start_kernel+0x4cb/0x58b
[ 2.735737] x86_64_start_reservations+0x24/0x26
[ 2.735936] x86_64_start_kernel+0x74/0x77
[ 2.736133] secondary_startup_64+0xa4/0xb0
[ 2.737333] ---[ end Kernel panic - not syncing: IO-APIC + timer doesn't work! Boot with apic=debug and send a report. Then try booting with the 'noapic' option. ]---
}}}

[Lee Yarwood (lyarwood)]

This is a duplicate of bug #1939108