Incorrect LAPIC/x2APIC parsing order

[Robert Malz]

[ Impact ]

 * In Kernels with commit ec9aedb2aa1a (6.7) CPUs can be enumerated in the order
   which violates ACPI specification that states:
    "OSPM should initialize processors in the order that they appear in the MADT"

 * The problematic commit parses all LAPIC entries before any x2APIC
   entries, aiming to ignore x2APIC entries with APIC ID < 255 when valid
   LAPIC entries exist. However, it disrupts the CPU enumeration order on
   systems where x2APIC entries precede LAPIC entries in the MADT.

 * In scenarios with strict CPU pinning this behavior might introduce
   performance degradation or hyperthreading related side-channel
   vulnerabilities

 * Issue has been fixed upstream with commit 0141978ae75

[ Test Plan ]

 * Issue reproduces only in specific ACPI MADT configurations with
   mixed LAPIC and x2APIC entries. An example of it would be AMD Genoa CPUs.

 * Easiest way to reproduce it locally is to replace x2APIC entry
   before LAPIC in MADT table.
   Steps to reproduce (local VM):
   1. Set up a VM with 288 vCPUs
    sudo qemu-system-x86_64 -enable-kvm \
      -M q35,accel=kvm,kernel-irqchip=split \
      -device intel-iommu,intremap=on \
      -smp 288,cores=144,threads=1,sockets=2 \
      -m 20G \
      -cpu host \
      -object memory-backend-ram,id=mem0,size=10G \
      -object memory-backend-ram,id=mem1,size=10G \
      -numa node,cpus=0-143,nodeid=0,memdev=mem0 \
      -numa node,cpus=144-287,nodeid=1,memdev=mem1 \
      -drive file=/var/lib/libvirt/images/ubuntu24.04.qcow2,format=qcow2 \
      -boot d
   2. Dump acpi apic/madt tables from VM
    acpidump > acpidump.dat
    acpixtract -a acpidump.dat
    iasl -d apic.dat
    ----
    apic.dsl file will be generated.
   3. Craft apic.dsl and switch order of CPUs
   4. Compile apic.dsl to aml file
    iasl -tc apic.dsl
   5. Inject new table to VM
    sudo qemu-system-x86_64 -enable-kvm \
      -M q35,accel=kvm,kernel-irqchip=split \
      -device intel-iommu,intremap=on \
      -smp 288,cores=144,threads=1,sockets=2 \
      -acpitable file=apic.aml \
      -m 20G \
      -cpu host \
      -object memory-backend-ram,id=mem0,size=10G \
      -object memory-backend-ram,id=mem1,size=10G \
      -numa node,cpus=0-143,nodeid=0,memdev=mem0 \
      -numa node,cpus=144-287,nodeid=1,memdev=mem1 \
      -drive file=/var/lib/libvirt/images/ubuntu24.04.qcow2,format=qcow2 \
      -boot d
    Results:
    Depending on MADT changes. For following example I have moved single x2APIC entry
    after the first LAPIC.
    Incorrect enumeration present on 6.8 Kernels:
    NUMA node(s): 2
    NUMA node0 CPU(s): 0-143
    NUMA node1 CPU(s): 144-287
    Enumeration on 6.8 Kernel with applied fixes:
    NUMA node(s): 2
    NUMA node0 CPU(s): 0,2-144
    NUMA node1 CPU(s): 1,145-287

[ Where problems could occur ]

 * Fix has been tested using alocal VM with a modified MADT table.

 * MADT table can have multiple possible configurations, making it hard to
   fully test the fix.

 * Potential regression with CPU enumeration can be hard to detect as was in case of
   patch that introduced the issue.

[ Other Info ]

 * Patch which introduced an issue: ec9aedb2aa1ab7ac420c00b31f5edc5be15ec167

 * Patch fixing the issue: 0141978ae75bd48bac13fca6de131a5071c32011

 * Originally issue has been reported for AMD Genoa CPU with following enumeration:
  6.5.0
   NUMA node(s): 2
   NUMA node0 CPU(s): 0-89,180-269
   NUMA node1 CPU(s): 90-179,270-359
  6.8.0
   NUMA node(s): 2
   NUMA node0 CPU(s): 0-179
   NUMA node1 CPU(s): 180-359

[Robert Malz (rmalz)]

attaching apic.dsl files

[Robert Malz (rmalz)]

modified apic.dsl

[Masahiro Yamada (myamada)]

Thank you for the report.

The commit has these tags:
  Cc: All applicable <email address hidden>
  Fixes: ec9aedb2aa1a ("x86/acpi: Ignore invalid x2APIC entries")

(https://github.com/torvalds/linux/commit/0141978ae75bd48bac13fca6de131a5071c32011)

So, I am pretty sure it will be back-ported to stable kernels.

Then, in Ubuntu kernels, we back-port fixes from stable kernels during regular release cycles called SRU.

This will be back-ported eventually.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.8.0-56.58 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux' to 'verification-done-noble-linux'. If the problem still exists, change the tag 'verification-needed-noble-linux' to 'verification-failed-noble-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/6.11.0-21.21 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oracular-linux' to 'verification-done-oracular-linux'. If the problem still exists, change the tag 'verification-needed-oracular-linux' to 'verification-failed-oracular-linux'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Robert Malz (rmalz)]

Tested CPU ordering on both 6.11.0-21-generic 6.8.0-56-generic with modified ACPI tables.
Everything looks correct:

Linux rmalz 6.11.0-19-generic #19~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Feb 17 11:51:52 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
NUMA node(s): 2
NUMA node0 CPU(s): 0-143
NUMA node1 CPU(s): 144-287

Linux rmalz 6.11.0-21-generic #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 19 16:50:40 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
NUMA node(s): 2
NUMA node0 CPU(s): 0,2-144
NUMA node1 CPU(s): 1,145-287

Linux rmalz 6.8.0-55-generic #57-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 12 23:42:21 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
NUMA node(s): 2
NUMA node0 CPU(s): 0-143
NUMA node1 CPU(s): 144-287

Linux rmalz 6.8.0-56-generic #58-Ubuntu SMP PREEMPT_DYNAMIC Fri Feb 14 15:33:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
NUMA node(s): 2
NUMA node0 CPU(s): 0,2-144
NUMA node1 CPU(s): 1,145-287

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-intel/6.11.0-1008.8 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-oracular-linux-intel' to 'verification-done-oracular-linux-intel'. If the problem still exists, change the tag 'verification-needed-oracular-linux-intel' to 'verification-failed-oracular-linux-intel'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra/6.8.0-1004.4 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-tegra' to 'verification-done-noble-linux-nvidia-tegra'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-tegra' to 'verification-failed-noble-linux-nvidia-tegra'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-tegra-pvw/6.8.0-1005.5 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-tegra-pvw' to 'verification-done-noble-linux-nvidia-tegra-pvw'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-tegra-pvw' to 'verification-failed-noble-linux-nvidia-tegra-pvw'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.8.0-56.58

---------------
linux (6.8.0-56.58) noble; urgency=medium

  * noble/linux: 6.8.0-56.58 -proposed tracker (LP: #2098244)

  * Noble update: upstream stable patchset 2024-07-19 (LP: #2073603)
    - Revert "drm: Make drivers depends on DRM_DW_HDMI"
    - Revert "UBUNTU: [Config] Drivers now depend on DRM_DW_HDMI"

  * drm/amd/display: Add check for granularity in dml ceil/floor helpers
    (LP: #2098080)
    - drm/amd/display: Add check for granularity in dml ceil/floor helpers

  * optimized default EPP for GNR family (LP: #2097554)
    - cpufreq: intel_pstate: Update Balance-performance EPP for Granite Rapids

  * Incorrect LAPIC/x2APIC parsing order (LP: #2097455)
    - x86/acpi: Fix LAPIC/x2APIC parsing order

  * MGLRU: page allocation failure on NUMA-enabled systems (LP: #2097214)
    - mm/vmscan: wake up flushers conditionally to avoid cgroup OOM

  * Upstream commit 65357e2c164a: "RDMA/mana_ib: set node_guid" applied
    incorrectly (LP: #2096885)
    - Revert "RDMA/mana_ib: set node_guid"

  * AppArmor early policy load not funcitoning (LP: #2095370)
    - SAUCE: Revert "UBUNTU: SAUCE: apparmor4.0.0 [67/90]: userns - add the
      ability to reference a global variable for a feature value"

  * apparmor unconfined profile blocks pivot_root (LP: #2067900)
    - SAUCE: Revert "UBUNTU: SAUCE: apparmor4.0.0 [81/90]: apparmor: convert easy
      uses of unconfined() to label_mediates()"

  * CVE-2024-50117
    - drm/amd: Guard against bad data for ATIF ACPI method

  * CVE-2024-56582
    - btrfs: fix use-after-free in btrfs_encoded_read_endio()

  * CVE-2024-53165
    - sh: intc: Fix use-after-free bug in register_intc_controller()

  * CVE-2024-53156
    - wifi: ath9k: add range check for conn_rsp_epid in htc_connect_service()

  * CVE-2024-56663
    - wifi: nl80211: fix NL80211_ATTR_MLO_LINK_ID off-by-one

  * CVE-2024-56614
    - xsk: fix OOB map writes when deleting elements

  * VM boots slowly with large-BAR GPU Passthrough due to pci/probe.c redundancy
    (LP: #2097389)
    - PCI: Batch BAR sizing operations

  * Noble update: upstream stable patchset 2025-02-04 (LP: #2097393)
    - Revert "PCI/MSI: Provide stubs for IMS functions"
    - gfs2: Revert "introduce qd_bh_get_or_undo"
    - gfs2: qd_check_sync cleanups
    - gfs2: Revert "ignore negated quota changes"
    - Revert "powerpc/ps3_defconfig: Disable PPC64_BIG_ENDIAN_ELF_ABI_V2"
    - tracing: Have saved_cmdlines arrays all in one allocation
    - spi: spi-fsl-lpspi: remove redundant spi_controller_put call
    - ata: ahci: Add mask_port_map module parameter
    - ASoC: tas2781: mark dvc_tlv with __maybe_unused
    - scsi: sd: Do not repeat the starting disk message
    - bootconfig: Fix the kerneldoc of _xbc_exit()
    - perf sched: Move start_work_mutex and work_done_wait_mutex initialization to
      perf_sched__replay()
    - perf sched: Fix memory leak in perf_sched__map()
    - perf sched: Move curr_thread initialization to perf_sched__map()
    - perf sched: Move curr_pid and cpu_last_switched initialization to
      perf_sched__{lat|map|replay}()
    - libsubcmd: Don't free the usage string
    - selftests: Introd...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 6.11.0-21.21

---------------
linux (6.11.0-21.21) oracular; urgency=medium

  * oracular/linux: 6.11.0-21.21 -proposed tracker (LP: #2098763)

  * Processes crash when attaching uretprobes to processes running in Docker
    (LP: #2098759)
    - seccomp: passthrough uretprobe systemcall without filtering

linux (6.11.0-20.20) oracular; urgency=medium

  * oracular/linux: 6.11.0-20.20 -proposed tracker (LP: #2098205)

  * drm/amd/display: Add check for granularity in dml ceil/floor helpers
    (LP: #2098080)
    - drm/amd/display: Add check for granularity in dml ceil/floor helpers

  * optimized default EPP for GNR family (LP: #2097554)
    - cpufreq: intel_pstate: Update Balance-performance EPP for Granite Rapids

  * Incorrect LAPIC/x2APIC parsing order (LP: #2097455)
    - x86/acpi: Fix LAPIC/x2APIC parsing order

  * MGLRU: page allocation failure on NUMA-enabled systems (LP: #2097214)
    - mm/vmscan: wake up flushers conditionally to avoid cgroup OOM

  * AppArmor early policy load not funcitoning (LP: #2095370)
    - SAUCE: Revert "UBUNTU: SAUCE: apparmor4.0.0 [66/99]: userns - add the
      ability to reference a global variable for a feature value"

  * apparmor unconfined profile blocks pivot_root (LP: #2067900)
    - SAUCE: Revert "UBUNTU: SAUCE: apparmor4.0.0 [80/99]: apparmor: convert easy
      uses of unconfined() to label_mediates()"

  * Oracular update: upstream stable patchset 2025-02-13 (LP: #2098165)
    - memblock: make memblock_set_node() also warn about use of MAX_NUMNODES
    - jbd2: increase IO priority for writing revoke records
    - jbd2: flush filesystem device before updating tail sequence
    - dm array: fix unreleased btree blocks on closing a faulty array cursor
    - dm array: fix cursor index when skipping across block boundaries
    - exfat: fix the infinite loop in __exfat_free_cluster()
    - ASoC: rt722: add delay time to wait for the calibration procedure
    - ASoC: mediatek: disable buffer pre-allocation
    - selftests/alsa: Fix circular dependency involving global-timer
    - ieee802154: ca8210: Add missing check for kfifo_alloc() in ca8210_probe()
    - net: 802: LLC+SNAP OID:PID lookup on start of skb data
    - tcp/dccp: allow a connection when sk_max_ack_backlog is zero
    - net: libwx: fix firmware mailbox abnormal return
    - pds_core: limit loop over fw name list
    - bnxt_en: Fix possible memory leak when hwrm_req_replace fails
    - cxgb4: Avoid removal of uninserted tid
    - ice: fix incorrect PHY settings for 100 GB/s
    - igc: return early when failing to read EECD register
    - tls: Fix tls_sw_sendmsg error handling
    - eth: gve: use appropriate helper to set xdp_features
    - Bluetooth: hci_sync: Fix not setting Random Address when required
    - Bluetooth: MGMT: Fix Add Device to responding before completing
    - Bluetooth: btnxpuart: Fix driver sending truncated data
    - Bluetooth: btnxpuart: Fix glitches seen in dual A2DP streaming
    - tcp: Annotate data-race around sk->sk_mark in tcp_v4_send_reset
    - netfilter: nf_tables: imbalance in flowtable binding
    - drm/mediatek: stop selecting foreign drivers
    - [Config] updat...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-nvidia-6.11/6.11.0-1005.5 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-nvidia-6.11' to 'verification-done-noble-linux-nvidia-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-nvidia-6.11' to 'verification-failed-noble-linux-nvidia-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-nvidia/6.8.0-1014.15 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-azure-nvidia' to 'verification-done-noble-linux-azure-nvidia'. If the problem still exists, change the tag 'verification-needed-noble-linux-azure-nvidia' to 'verification-failed-noble-linux-azure-nvidia'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-6.11/6.11.0-1013.14~24.04.1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-aws-6.11' to 'verification-done-noble-linux-aws-6.11'. If the problem still exists, change the tag 'verification-needed-noble-linux-aws-6.11' to 'verification-failed-noble-linux-aws-6.11'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-raspi/6.8.0-1028.32 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-raspi' to 'verification-done-noble-linux-raspi'. If the problem still exists, change the tag 'verification-needed-noble-linux-raspi' to 'verification-failed-noble-linux-raspi'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-fips/6.8.0-72.72+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-fips' to 'verification-done-noble-linux-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-fips' to 'verification-failed-noble-linux-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-gcp-fips/6.8.0-1035.37+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-gcp-fips' to 'verification-done-noble-linux-gcp-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-gcp-fips' to 'verification-failed-noble-linux-gcp-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-aws-fips/6.8.0-1034.36+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-aws-fips' to 'verification-done-noble-linux-aws-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-aws-fips' to 'verification-failed-noble-linux-aws-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-xilinx/6.8.0-1017.18 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-xilinx' to 'verification-done-noble-linux-xilinx'. If the problem still exists, change the tag 'verification-needed-noble-linux-xilinx' to 'verification-failed-noble-linux-xilinx'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-azure-fips/6.8.0-1034.39+fips1 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-noble-linux-azure-fips' to 'verification-done-noble-linux-azure-fips'. If the problem still exists, change the tag 'verification-needed-noble-linux-azure-fips' to 'verification-failed-noble-linux-azure-fips'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!