ice/iavf: drivers panic during shutdown when "reboot -f"

[Jiping Ma]

This issue can be reproduced with the below steps.
1. Installed sts-silicom app.
2. Make sure sts-silicom must be running status.
3. reboot -f

[ 942.346200] general protection fault, probably for non-canonical address 0xc0000000009082e: 0000 [#1] PREEMPT SMP NOPTI
[ 942.356980] CPU: 0 PID: 162411 Comm: reboot Kdump: loaded Tainted: G S O 5.10.0-6-amd64 #1 Debian 5.10.162-1.stx.64
[ 942.356981] Hardware name: Dell Inc. PowerEdge XR11/0P2RNT, BIOS 1.9.2 11/17/2022
[ 942.356986] RIP: 0010:_raw_spin_lock_irqsave+0x19/0x40
[ 942.356988] Code: 89 e0 41 5c c3 cc cc cc cc 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 41 54 9c 41 5c fa 65 ff 05 ce 0d f9 63 31 c0 ba 01 00 00 00 <f0> 0f b1 17 75 0a 4c 89 e0 41 5c c3 cc cc cc cc 89 c6 e8 30 c5 69
[ 942.356989] RSP: 0018:ff29f07130ab7ad0 EFLAGS: 00010046
[ 942.356992] RAX: 0000000000000000 RBX: 0000000022223a22 RCX: 0000000000000000
[ 942.412272] RDX: 0000000000000001 RSI: ff29f07130ab7b18 RDI: 0c0000000009082e
[ 942.412273] RBP: ff28ecaf18ae8980 R08: 0000000000000000 R09: ffffffff9d0478b8
[ 942.412274] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000046
[ 942.433669] R13: ff29f07130ab7b18 R14: 0000000000023e40 R15: 0c0000000009082e
[ 942.433670] FS: 00007fb31bca88c0(0000) GS:ff28eccd7e600000(0000) knlGS:0000000000000000
[ 942.433671] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 942.433672] CR2: 000056412ccc8000 CR3: 00000003b9aee003 CR4: 0000000000771ef0
[ 942.433673] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 942.468899] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 942.468899] PKRU: 55555554
[ 942.468900] Call Trace:
[ 942.481202] lock_timer_base+0x61/0x80
[ 942.484960] del_timer+0x33/0x80
[ 942.488191] try_to_grab_pending+0xca/0x170
[ 942.492376] __cancel_work_timer+0x3c/0x190
[ 942.496564] ? kernfs_put.part.0+0xd9/0x1a0
[ 942.500749] ? kmem_cache_free+0x3bd/0x410
[ 942.504851] iavf_remove+0x6a/0xe0 [iavf]
[ 942.508861] ? pci_device_remove+0x38/0xa0
[ 942.512961] ? __device_release_driver+0x17b/0x250
[ 942.517753] ? device_release_driver+0x24/0x30
[ 942.522197] ? pci_stop_bus_device+0x6c/0x90
[ 942.526470] ? pci_stop_and_remove_bus_device+0xe/0x20
[ 942.531613] ? pci_iov_remove_virtfn+0xc0/0x130
[ 942.536144] ? sriov_disable+0x34/0xe0
[ 942.539928] ? ice_free_vfs+0x77/0x350 [ice]
[ 942.544206] ? flow_indr_dev_unregister+0x243/0x250
[ 942.549104] ? ice_remove+0x3e5/0x430 [ice]
[ 942.553293] ? ice_shutdown+0x16/0x50 [ice]
[ 942.557481] ? pci_device_shutdown+0x31/0x60
[ 942.561755] ? device_shutdown+0x156/0x1b0
[ 942.565854] ? __do_sys_reboot.cold+0x2f/0x5b
[ 942.570212] ? vfs_writev+0xc5/0x160
[ 942.573793] ? do_syscall_64+0x30/0x40
[ 942.577545] ? entry_SYSCALL_64_after_hwframe+0x61/0xc6
[ 942.582769] Modules linked in: ip6_tables ipt_rpfilter ip6t_rpfilter xt_set xt_multiport ip_set_hash_ip ip_set_hash_net veth wireguard curve25519_x86_64 libcurve25519_generic libchacha20poly1305 chacha_x86_64 poly1305_x86_64 ip6_udp_tunnel udp_tunnel libchacha nbd rbd libceph dns_resolver nf_conntrack_netlink xt_nat ip6t_REJECT nf_reject_ipv6 ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_conntrack xt_MASQUERADE nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_comment xt_mark vfio_pci vfio_virqfd binfmt_misc drbd virtio_net net_failover failover nft_counter xt_addrtype nft_compat nf_tables nfnetlink br_netfilter bridge stp llc nfsd auth_rpcgss nfs_acl lockd grace overlay xfs vfio_iommu_type1 vfio oct_ep_phc(O) xprtrdma(O) svcrdma(O) rpcrdma(O) nvmet_rdma(O) nvme_rdma(O) ib_srp(O) ib_isert(O) ib_iser(O) rdma_rxe(O) mlx5_ib(O) mlx5_core(O) mlxfw(O) mlxdevm(O) psample tls pci_hyperv_intf rdma_ucm(O) rdma_cm(O) iw_cm(O) ib_uverbs(O) ib_ucm(O) ib_cm(O) lru_cache libcrc32c fuse
[ 942.582819] drm sunrpc bpf_preload efivarfs ip_tables ext4 mbcache jbd2 dm_multipath dm_mod sd_mod t10_pi iTCO_wdt iTCO_vendor_support bnxt_re(O) x86_pkg_temp_thermal intel_powerclamp coretemp ib_core(O) kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul mlx_compat(O) crc32c_intel ghash_clmulni_intel rapl intel_uncore acpi_ipmi ipmi_si bnxt_en(O) i2c_i801 i2c_smbus ipmi_devintf mei_me cdc_acm ahci libahci ftdi_sio mei ipmi_msghandler tpm_crb tpm_tis tpm_tis_core tpm acpi_power_meter iavf(O) i40e(O) ice(O) [last unloaded: drbd]

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/starlingx/kernel/+/893311

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/c/starlingx/kernel/+/893311
Committed: https://opendev.org/starlingx/kernel/commit/2a5f9f349051759948833030475db54f78516b93
Submitter: "Zuul (22348)"
Branch: master

commit 2a5f9f349051759948833030475db54f78516b93
Author: Jiping Ma <email address hidden>
Date: Tue Aug 8 00:20:22 2023 -0700

    kernel: Fix drivers panic during shutdown

    This commit prevents the ice and iavf drivers from causing kernel
    panics when forced reboot is initiated with "reboot -f".

    Issue #1: iavf driver

    If the netdev pointer is NULL, then iavf_remove() returns early to
    ensure that it does not proceed with an already-freed netdev instance.
    However, drvdata field of the iavf driver's pci_dev structure continues
    to keep the former value of the netdev pointer, and this value can be
    acquired from the pci_dev structure via pci_get_drvdata(). This causes
    a kernel panic when a forced reboot/shutdown is in progress due to the
    following sequence of events:

    - The iavf_shutdown() callback is called by the kernel. This function
      detaches the device, brings it down if it was running and frees
      resources.
    - Later, the associated PF driver's shutdown callback is called:
      ice_shutdown(). That callback calls, among others, sriov_disable(),
      which then indirectly calls iavf_remove() again.
    - Kernel WARNING is reported because the work adminq_task->func is NULL
      in cancel_work_sync(&adapter->adminq_task) during iavf_remove(), that
      reason is the resource already had been freed in the first
      iavf_remove() running stage.
      "WARNING: CPU: 63 PID: 93678 at kernel/workqueue.c:3047
        __flush_work.isra.0+0x6b/0x80"

    The patch for iavf resolves this issue by checking the pci_dev
    structure's is_busmaster field at the beginning of iavf_remove(). If the
    PCI device had already been disabled by an earlier call to
    iavf_shutdown() or iavf_remove(), via a call to pci_disable_device(),
    then the is_busmaster field would be set to zero. Based on this logic,
    if the is_busmaster field is set to zero, then the iavf_remove function
    returns early. This in turn avoids the aforementioned kernel panic
    caused by multiple calls to iavf_remove().

    Note that the description above is applicable to iavf-4.6.1 (in NIC
    driver bundle cvl-4.10); however, a similar issue occurs in earlier
    versions of the iavf driver as well, which necessitates the same fix.

    Issue #2: ice driver

    When the system is rebooted, then the PTP-related resources are released
    by the ice driver's ice_remove() function before the irq_msix_misc
    interrupt is disabled. However, the interrupt handler continues to use
    these resources, and when the interrupt in question occurs, then a
    kernel panic occurs.

    This issue is fixed by disabling the irq_msix_misc interrupt before the
    call to ice_ptp_release() in ice_remove().

    Please note that colleagues at Intel have reviewed the fixes included in
    this commit, and they have confirmed that these changes could be used as
    a temporary workaround ...