System freeze after resuming from suspend due to PCI ASPM settings

[AceLan Kao]

For OEM-6.1
[Impact]
While doing some tests such as suspend/resume or CPU stress tests the system would hang.

[Fix]
Below commit fixed the issue, but not going to be merged into mainline.
The patch is still under discussion and have other variance, and we already merged the origin patch into oem-6.0 and 5.15/5.19 for a year, so could consider it's safer for us.
https://patchwork.<email address hidden>/

I also created a DMI quirk to make the patches only affects on listed platforms.

[Test]
The affected machines could suspend/resume well.

[Where problems could occur]
The patches only affects on the listed platforms, and won't affect other platforms.

======================================================================
For Jammy/Kinetic SRU

[Impact]
While doing some tests such as suspend/resume or CPU stress tests the system would hang.

[Fix]
The 2 commits fix the issue, but still not get accepted yet.
https://patchwork.<email address hidden>/
https://patchwork.ozlabs.org<email address hidden>/

So, I created a DMI quirk to make the patches only affects on listed platforms.

[Test]
Verified on the failed machines and ODM also verified on their side.

[Where problems could occur]
The patches only affects on the listed platforms, and won't affect other platforms.

======================================================================
For OEM-6.0

[Impact]
While doing some tests such as suspend/resume or CPU stress tests the system would hang.

[Fix]
The 2 commits fix the issue, but still not get accepted yet.
https://patchwork.<email address hidden>/
https://patchwork.ozlabs.org<email address hidden>/

[Test]
Verified on the failed machines and ODM also verified on their side.

[Where problems could occur]
The 2 patches look pretty safe to me, they try to preserve the ASPM state of devices.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1980829

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oem-5.14/5.14.0-1046.53 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-oem-5.17/5.17.0-1014.15 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-oem-5.14 - 5.14.0-1046.53

---------------
linux-oem-5.14 (5.14.0-1046.53) focal; urgency=medium

  * focal/linux-oem-5.14: 5.14.0-1046.53 -proposed tracker (LP: #1980928)

  * alsa: asoc: amd: the internal mic can't be dedected on yellow carp machines
    (LP: #1980700)
    - ASoC: amd: Add driver data to acp6x machine driver
    - ASoC: amd: Add support for enabling DMIC on acp6x via _DSD

  * CVE-2022-34918
    - netfilter: nf_tables: stricter validation of element data

  * System freeze after resuming from suspend due to PCI ASPM settings
    (LP: #1980829)
    - PCI/ASPM: Save/restore L1SS Capability for suspend/resume
    - PCI:ASPM: Remove pcie_aspm_pm_state_change()

 -- Chia-Lin Kao (AceLan) <email address hidden> Wed, 13 Jul 2022 21:02:35 +0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-oem-5.17 - 5.17.0-1014.15

---------------
linux-oem-5.17 (5.17.0-1014.15) jammy; urgency=medium

  * jammy/linux-oem-5.17: 5.17.0-1014.15 -proposed tracker (LP: #1981244)

  * Clear PCI errors left from BIOS (LP: #1981173)
    - PCI: Clear PCI_STATUS when setting up device

  * intel_iommu: Fix enable intel_iommu, Ubuntu 22.04 installation crashes
    (LP: #1982104)
    - iommu/vt-d: Fix RID2PASID setup/teardown failure

  * Failed to resume from S3 blocked by atlantic driver[1d6a:94c0]
    (LP: #1981950)
    - net: atlantic: remove deep parameter on suspend/resume functions
    - net: atlantic: remove aq_nic_deinit() when resume

  * Make cm32181 sensor work after system suspend (LP: #1981773)
    - iio: light: cm32181: Add PM support

  * alsa: asoc: amd: the internal mic can't be dedected on yellow carp machines
    (LP: #1980700)
    - ASoC: amd: Add support for enabling DMIC on acp6x via _DSD

  * CVE-2022-34918
    - netfilter: nf_tables: stricter validation of element data

  * System freeze after resuming from suspend due to PCI ASPM settings
    (LP: #1980829)
    - PCI/ASPM: Save/restore L1SS Capability for suspend/resume
    - PCI:ASPM: Remove pcie_aspm_pm_state_change()

 -- Chia-Lin Kao (AceLan) <email address hidden> Tue, 19 Jul 2022 22:07:45 +0800

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.15.0-48.54

---------------
linux (5.15.0-48.54) jammy; urgency=medium

  * jammy/linux: 5.15.0-48.54 -proposed tracker (LP: #1987775)

  * System freeze after resuming from suspend due to PCI ASPM settings
    (LP: #1980829)
    - SAUCE: PCI/ASPM: Save/restore L1SS Capability for suspend/resume
    - SAUCE: whitelist platforms that needs save/restore ASPM L1SS for
      suspend/resume

  * [SRU][J/OEM-5.17][PATCH 0/1] Fix oled brightness set above frame-average
    luminance (LP: #1978986)
    - SAUCE: drm: New function to get luminance range based on static hdr metadata
    - SAUCE: drm/amdgpu_dm: Rely on split out luminance calculation function
    - SAUCE: drm/i915: Use luminance range calculated during edid parsing

  * Jammy: Add OVS Internal Port HW Offload to mlx5 driver (LP: #1983498)
    - net/mlx5e: Refactor rx handler of represetor device
    - net/mlx5e: Use generic name for the forwarding dev pointer
    - net/mlx5: E-Switch, Add ovs internal port mapping to metadata support
    - net/mlx5e: Support accept action
    - net/mlx5e: Accept action skbedit in the tc actions list
    - net/mlx5e: Offload tc rules that redirect to ovs internal port
    - net/mlx5e: Offload internal port as encap route device
    - net/mlx5e: Enable TC offload for ingress MACVLAN
    - net/mlx5e: Add indirect tc offload of ovs internal port
    - net/mlx5e: Term table handling of internal port rules
    - net/mlx5: Support internal port as decap route device
    - net/mlx5: Fix some error handling paths in 'mlx5e_tc_add_fdb_flow()'
    - net/mlx5e: TC, Fix memory leak with rules with internal port
    - net/mlx5e: Fix skb memory leak when TC classifier action offloads are
      disabled
    - net/mlx5e: Fix nullptr on deleting mirroring rule
    - net/mlx5e: Avoid implicit modify hdr for decap drop rule
    - net/mlx5e: Fix wrong source vport matching on tunnel rule
    - net/mlx5e: TC, fix decap fallback to uplink when int port not supported

  * Remove unused variable from i915 psr (LP: #1986798)
    - SAUCE: drm/i915/display/psr: Remove unused variable

  * refactoring of overlayfs fix to properly support shiftfs (LP: #1983640)
    - SAUCE: overlayfs: remove CONFIG_AUFS_FS dependency

  * Jammy update: v5.15.53 upstream stable release (LP: #1986728)
    - Revert "drm/amdgpu/display: set vblank_disable_immediate for DC"
    - drm/amdgpu: To flush tlb for MMHUB of RAVEN series
    - ksmbd: set the range of bytes to zero without extending file size in
      FSCTL_ZERO_DATA
    - ksmbd: check invalid FileOffset and BeyondFinalZero in FSCTL_ZERO_DATA
    - ksmbd: use vfs_llseek instead of dereferencing NULL
    - ipv6: take care of disable_policy when restoring routes
    - net: phy: Don't trigger state machine while in suspend
    - nvme-pci: add NVME_QUIRK_BOGUS_NID for ADATA XPG SX6000LNP (AKA SPECTRIX
      S40G)
    - nvme-pci: add NVME_QUIRK_BOGUS_NID for ADATA IM2P33F8ABR1
    - nvdimm: Fix badblocks clear off-by-one error
    - powerpc/prom_init: Fix kernel config grep
    - powerpc/book3e: Fix PUD allocation size in map_kernel_page()
    - powerpc/bpf: Fix use of user_pt_regs in uapi
    - dm raid: fix ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.19.0-18.18

---------------
linux (5.19.0-18.18) kinetic; urgency=medium

  * kinetic/linux: 5.19.0-18.18 -proposed tracker (LP: #1990366)

  * 5.19.0-17.17: kernel NULL pointer dereference, address: 0000000000000084
    (LP: #1990236)
    - Revert "UBUNTU: SAUCE: apparmor: Fix regression in stacking due to label
      flags"
    - Revert "UBUNTU: [Config] disable SECURITY_APPARMOR_RESTRICT_USERNS"
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - add an internal buffer""
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - don't wait on cleanup""
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - don't waste entropy""
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - always add a pending
      request""
    - Revert "UBUNTU: SAUCE: Revert "hwrng: virtio - unregister device before
      reset""
    - Revert "UBUNTU: SAUCE: Revert "virtio-rng: make device ready before making
      request""
    - Revert "UBUNTU: [Config] update configs after apply new apparmor patch set"
    - Revert "UBUNTU: SAUCE: apparmor: add user namespace creation mediation"
    - Revert "UBUNTU: SAUCE: selinux: Implement userns_create hook"
    - Revert "UBUNTU: SAUCE: bpf-lsm: Make bpf_lsm_userns_create() sleepable"
    - Revert "UBUNTU: SAUCE: security, lsm: Introduce security_create_user_ns()"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: AppArmor: Remove the exclusive
      flag"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Add /proc attr entry for full
      LSM context"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Removed scaffolding function
      lsmcontext_init"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: netlabel: Use a struct lsmblob in
      audit data"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Add record for multiple
      object contexts"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: audit: multiple subject lsm values
      for netlabel"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Add record for multiple task
      security contexts"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Allow multiple records in an
      audit_buffer"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Add a function to report
      multiple LSMs"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Create audit_stamp
      structure"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: Audit: Keep multiple LSM data in
      audit_names"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: security_secid_to_secctx
      module selection"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: binder: Pass LSM identifier for
      confirmation"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: NET: Store LSM netlabel data in a
      lsmblob"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: security_secid_to_secctx in
      netlink netfilter"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Use lsmcontext in
      security_dentry_init_security"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Use lsmcontext in
      security_inode_getsecctx"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM: Use lsmcontext in
      security_secid_to_secctx"
    - Revert "UBUNTU: SAUCE: lsm stacking v37: LSM:...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-bluefield/5.15.0-1010.12 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy' to 'verification-done-jammy'. If the problem still exists, change the tag 'verification-needed-jammy' to 'verification-failed-jammy'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Timo Aaltonen (tjaalton)]

please verify oem-6.1 1020.20

[AceLan Kao (acelankao)]

No machine to verify the issue, so only checked the code has been merged.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux-oem-6.1 - 6.1.0-1020.20

---------------
linux-oem-6.1 (6.1.0-1020.20) jammy; urgency=medium

  * jammy/linux-oem-6.1: 6.1.0-1020.20 -proposed tracker (LP: #2030594)

  * CVE-2022-40982
    - init: Provide arch_cpu_finalize_init()
    - x86/cpu: Switch to arch_cpu_finalize_init()
    - ARM: cpu: Switch to arch_cpu_finalize_init()
    - ia64/cpu: Switch to arch_cpu_finalize_init()
    - loongarch/cpu: Switch to arch_cpu_finalize_init()
    - m68k/cpu: Switch to arch_cpu_finalize_init()
    - mips/cpu: Switch to arch_cpu_finalize_init()
    - sh/cpu: Switch to arch_cpu_finalize_init()
    - sparc/cpu: Switch to arch_cpu_finalize_init()
    - um/cpu: Switch to arch_cpu_finalize_init()
    - init: Remove check_bugs() leftovers
    - init: Invoke arch_cpu_finalize_init() earlier
    - init, x86: Move mem_encrypt_init() into arch_cpu_finalize_init()
    - x86/init: Initialize signal frame size late
    - x86/fpu: Remove cpuinfo argument from init functions
    - x86/fpu: Mark init functions __init
    - x86/fpu: Move FPU initialization into arch_cpu_finalize_init()
    - x86/speculation: Add Gather Data Sampling mitigation
    - x86/speculation: Add force option to GDS mitigation
    - x86/speculation: Add Kconfig option for GDS
    - KVM: Add GDS_NO support to KVM
    - x86/mem_encrypt: Unbreak the AMD_MEM_ENCRYPT=n build
    - x86/xen: Fix secondary processors' FPU initialization
    - x86/mm: fix poking_init() for Xen PV guests
    - x86/mm: Use mm_alloc() in poking_init()
    - mm: Move mm_cachep initialization to mm_init()
    - x86/mm: Initialize text poking earlier
    - Documentation/x86: Fix backwards on/off logic about YMM support
    - [Config]: Enable CONFIG_ARCH_HAS_CPU_FINALIZE_INIT

  * System freeze after resuming from suspend due to PCI ASPM settings
    (LP: #1980829)
    - SAUCE: PCI/ASPM: Save/restore L1SS Capability for suspend/resume
    - SAUCE: whitelist platforms that needs save/restore ASPM L1SS for
      suspend/resume

  * CVE-2023-20593
    - x86/cpu/amd: Move the errata checking functionality up
    - x86/cpu/amd: Add a Zenbleed fix

  * Fix repeated errors of blacklisting during bootup (LP: #2029363)
    - certs: make blacklisted hash available in klog
    - KEYS: Add new function key_create()
    - certs: don't try to update blacklist keys

  * Fix AMD gpu hang when screen off/on (LP: #2028740)
    - drm/amd/display: Keep PHY active for dp config

  * CVE-2023-4015
    - netfilter: nf_tables: skip immediate deactivate in _PREPARE_ERROR

  * CVE-2023-3995
    - netfilter: nf_tables: disallow rule addition to bound chain via
      NFTA_RULE_CHAIN_ID

  * CVE-2023-3777
    - netfilter: nf_tables: skip bound chain on rule flush

  * CVE-2023-4004
    - netfilter: nft_set_pipapo: fix improper element removal

 -- Timo Aaltonen <email address hidden> Wed, 16 Aug 2023 15:20:53 +0300

[AceLan Kao (acelankao)]

The final discuss and version are here

https://bugzilla.kernel.org/show_bug.cgi?id=216877
https://patchwork.kernel<email address hidden>/

And koba has backported the old version of that commit to oem kernel
commit 16e5386dabd18bd9c507867b0df6c414783af4a8
Author: Mika Westerberg <email address hidden>
Date: Mon Oct 2 10:00:44 2023 +0300

    UBUNTU: SAUCE: PCI/ASPM: Add back L1 PM Substate save and restore

    BugLink: https://bugs.launchpad.net/bugs/2042500

    Commit a7152be79b62 ("Revert "PCI/ASPM: Save L1 PM Substates Capability
    for suspend/resume"") reverted saving and restoring of ASPM L1 Substates
    due to a regression that caused resume from suspend to fail on certain
    systems. However, we never added this capability back and this is now
    causing systems fail to enter low power CPU states, drawing more power
    from the battery.

    The original revert mentioned that we restore L1 PM substate configuration
    even though ASPM L1 may already be enabled. This is due the fact that
    the pci_restore_aspm_l1ss_state() was called before pci_restore_pcie_state().

    Try to enable this functionality again following PCIe r6.0.1, sec 5.5.4
    more closely by:

      1) Do not restore ASPM configuration in pci_restore_pcie_state() but
         do that after PCIe capability is restored in pci_restore_aspm_state()
         following PCIe r6.0, sec 5.5.4.

      2) ASPM is first enabled on the upstream component and then downstream
         (this is already forced by the parent-child ordering of Linux
         Device Power Management framework).

      3) Program ASPM L1 PM substate configuration before L1 enables.

      4) Program ASPM L1 PM substate enables last after rest of the fields
         in the capability are programmed.

      5) Add denylist that skips restoring on the ASUS and TUXEDO systems
         where these regressions happened, just in case. For the TUXEDO case
         we only skip restore if the BIOS is involved in system suspend
         (that's forcing "mem_sleep=deep" in the command line). This is to
         avoid possible power regression when the default suspend to idle is
         used, and at the same time make sure the devices continue working
         after resume when the BIOS is involved.

    Reported-by: Koba Ko <email address hidden>
    Closes: https://bugzilla.kernel.org/show_bug.cgi?id=217321
    Link: https://bugzilla.kernel.org/show_bug.cgi?id=216782
    Link: https://bugzilla.kernel.org/show_bug.cgi?id=216877
    Cc: Tasev Nikola <email address hidden>
    Cc: Mark Enriquez <email address hidden>
    Cc: Thomas Witt <email address hidden>
    Cc: Werner Sembach <email address hidden>
    Tested-by: Kai-Heng Feng <email address hidden>
    Signed-off-by: Mika Westerberg <email address hidden>
    Reviewed-by: Ilpo Järvinen <email address hidden>
    (backported from https://<email address hidden>/)
    Signed-off-by: Koba Ko <email address hidden>
    Signed-off-by: Timo Aaltonen <timo...

[Anthony Wong (anthonywong)]

Same as LP#2042500

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux-mtk/5.15.0-1030.34 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-done-jammy-linux-mtk'. If the problem still exists, change the tag 'verification-needed-jammy-linux-mtk' to 'verification-failed-jammy-linux-mtk'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!