kernel bug found when disconnecting one fiber channel interface on Cisco Chassis with fnic DRV_VERSION "1.6.0.47"

[Eric Desrochers]

[Impact]

It has been brought to my attention the following:

"
We have been experiencing node lockups and degradation when testing fiber channel fail over for multi-path PURESTORAGE drives.

Testing usually consists of either failing over the fabric or the local I/O module for the Cisco chassis which houses a number of individual blades.

After rebooting a local Chassis I/O module we see commands like multipath -ll hanging.
Resetting the blades individual fiber channel interface results in the following messages.
"

6051160.241383] rport-9:0-1: blocked FC remote port time out: removing target and saving binding
[6051160.252901] BUG: kernel NULL pointer dereference, address: 0000000000000040
[6051160.262267] #PF: supervisor read access in kernel mode
[6051160.269314] #PF: error_code(0x0000) - not-present page
[6051160.276016] PGD 0 P4D 0
[6051160.279807] Oops: 0000 [#1] SMP NOPTI
[6051160.284642] CPU: 10 PID: 49346 Comm: kworker/10:2 Tainted: P O 5.4.0-77-generic #86-Ubuntu
[6051160.295967] Hardware name: Cisco Systems Inc UCSB-B200-M5/UCSB-B200-M5, BIOS B200M5.4.1.1d.0.0609200543 06/09/2020
[6051160.308199] Workqueue: fc_dl_9 fc_timeout_deleted_rport [scsi_transport_fc]
[6051160.316640] RIP: 0010:fnic_terminate_rport_io+0x10f/0x510 [fnic]
[6051160.324050] Code: 48 89 c3 48 85 c0 0f 84 7b 02 00 00 48 05 20 01 00 00 48 89 45 b0 0f 84 6b 02 00 00 48 8b 83 58 01 00 00 48 8b 80 b8 01 00 00 <48> 8b 78 40 e8 68 e6 06 00 85 c0 0f 84 4c 02 00 00 48 8b 83 58 01
[6051160.346553] RSP: 0018:ffffbc224f297d90 EFLAGS: 00010082
[6051160.353115] RAX: 0000000000000000 RBX: ffff90abdd4c4b00 RCX: ffff90d8ab2c2bb0
[6051160.361983] RDX: ffff90d8b5467400 RSI: 0000000000000000 RDI: ffff90d8ab3b4b40
[6051160.370812] RBP: ffffbc224f297df8 R08: ffff90d8c08978c8 R09: ffff90d8b8850800
[6051160.379518] R10: ffff90d8a59d64c0 R11: 0000000000000001 R12: ffff90d8ab2c31f8
[6051160.388242] R13: 0000000000000000 R14: 0000000000000246 R15: ffff90d8ab2c27b8
[6051160.396953] FS: 0000000000000000(0000) GS:ffff90d8c0880000(0000) knlGS:0000000000000000
[6051160.406838] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[6051160.414168] CR2: 0000000000000040 CR3: 0000000fc1c0a004 CR4: 00000000007626e0
[6051160.423146] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[6051160.431884] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[6051160.440615] PKRU: 55555554
[6051160.444337] Call Trace:
[6051160.447841] fc_terminate_rport_io+0x56/0x70 [scsi_transport_fc]
[6051160.455263] fc_timeout_deleted_rport.cold+0x1bc/0x2c7 [scsi_transport_fc]
[6051160.463623] process_one_work+0x1eb/0x3b0
[6051160.468784] worker_thread+0x4d/0x400
[6051160.473660] kthread+0x104/0x140
[6051160.478102] ? process_one_work+0x3b0/0x3b0
[6051160.483439] ? kthread_park+0x90/0x90
[6051160.488213] ret_from_fork+0x1f/0x40
[6051160.492901] Modules linked in: dm_service_time zfs(PO) zunicode(PO) zlua(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) ebtable_filter ebtables ip6table_raw ip6table_mangle ip6table_nat iptable_raw iptable_mangle iptable_nat nf_nat vhost_vsock vmw_vsock_virtio_transport_common vsock unix_diag nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 vhost_net vhost tap 8021q garp mrp bluetooth ecdh_generic ecc tcp_diag inet_diag sctp nf_tables nfnetlink ip6table_filter ip6_tables iptable_filter bpfilter bridge stp llc nls_iso8859_1 dm_queue_length dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua intel_rapl_msr intel_rapl_common isst_if_common skx_edac nfit x86_pkg_temp_thermal intel_powerclamp ipmi_ssif coretemp kvm_intel kvm rapl input_leds joydev intel_cstate mei_me ioatdma mei dca ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad mac_hid sch_fq_codel ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
[6051160.492928] async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear fnic mgag200 drm_vram_helper i2c_algo_bit ttm drm_kms_helper crct10dif_pclmul syscopyarea hid_generic crc32_pclmul libfcoe sysfillrect ghash_clmulni_intel sysimgblt aesni_intel fb_sys_fops crypto_simd libfc usbhid cryptd scsi_transport_fc hid drm glue_helper enic ahci lpc_ich libahci wmi
[6051160.632623] CR2: 0000000000000040
[6051160.637043] ---[ end trace 236e6f4850146477 ]---

[Test Plan]

There are two ways to replicate the bug:

Reset a single chassis I/O module or fail over a fabric interconnect (FI) for all chassis in the cluster. We have performed both tests.

Specific hardware:
   Chassis Cisco UCS 5108 AC2 Chassis
   Blades Cisco UCS B200
   IO module Cisco UCS 2408

Server loads - Ubuntu 20.04 cluster running deployed maas, juju and openstack.

Tests

1) Fail over of single chassis I/O module results in at least one node locking up.
After patching the kernel multiple tests by resetting the I/O module did not result in further failures. Each chassis holds 8 blades.

2) The larger test reboots the actual FI (fabric interconnect) for one channel serving 3 chassis. What we call a fiber channel fail over test.
This test covers 3 chassis with 8 blades each. In this test at least one and often as many as 4 nodes will lock up. After loading the patched kernel we ran this test 3 times with no failures.

[Where problems could occur]

Cisco "fNIC" driver enables FCoE support for the Cisco UCS Virtual Interface Card family of products.

If a problem arise it would be limited to these VIC which are specially designed for Cisco UCS blade and rack servers and possibly command to terminate I/O in any case at worst case (again only on Cisco UCS hw family.

Note that Field Engineer and I did test the patch on Cisco UCS hw and the patch didn't reproduce the problem nor produce observable subsequent issues/regressions.

[Other informations]

https://support.oracle.com/knowledge/Oracle%20Linux%20and%20Virtualization/2792832_1.html#FIX
https://www.spinics.net/lists/linux-scsi/msg142179.html

[Eric Desrochers (slashd)]

[Potential fix candidate]

commit 712582e60f288e7cede8d6fc8769529317e0f3e0
Author: Hannes Reinecke <email address hidden>
Date: Fri May 15 13:26:47 2020 +0200

scsi: fnic: Do not call 'scsi_done()' for unhandled commands

The fnic drivers assigns an ioreq structure to each command and severs this
assignment once scsi_done() has been called and the command has been
completed.

When traversing commands to terminate outstanding I/O we should not call
scsi_done() on commands which do not have a corresponding ioreq structure;
these commands have either never entered the driver or have already been
completed.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1944586

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Eric Desrochers (slashd)]

A test kernel of v5.4 (kernel series where the problem has been found) has been tested by Field Engineer and here's the outcome:

"
-Extensive testing about 4/5 failovers both HWE (v5.11) and the patched kernels seem stable (v5.4).

Thank you this unblocks us for deployment of this cloud.
"

- Eric

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is awaiting verification that the linux/5.4.0-90.101 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[Kelsey Steele (kelsey-steele)]

Hi Eric, may you please verify the focal kernel in -proposed resolves this bug? You can find more instructions for this in comment #4. Thank you!

[Eric Desrochers (slashd)]

This has been tested on Cisco Hardware by Field Engineering and the bug is no longer reproducible.

- Eric

[Kelsey Steele (kelsey-steele)]

Thank you, Eric! :)

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 5.4.0-90.101

---------------
linux (5.4.0-90.101) focal; urgency=medium

  * focal/linux: 5.4.0-90.101 -proposed tracker (LP: #1947260)

  * Packaging resync (LP: #1786013)
    - debian/dkms-versions -- update from kernel-versions (main/2021.10.18)

  * Add final-checks to check certificates (LP: #1947174)
    - [Packaging] Add system trusted and revocation keys final check

  * No sound on Lenovo laptop models Legion 15IMHG05, Yoga 7 14ITL5, and 13s
    Gen2 (LP: #1939052)
    - ALSA: hda/realtek: Quirks to enable speaker output for Lenovo Legion 7i
      15IMHG05, Yoga 7i 14ITL5/15ITL5, and 13s Gen2 laptops.
    - ALSA: hda/realtek: Fix for quirk to enable speaker output on the Lenovo 13s
      Gen2

  * CVE-2020-36385
    - RDMA/cma: Add missing locking to rdma_accept()
    - RDMA/ucma: Fix the locking of ctx->file
    - RDMA/ucma: Rework ucma_migrate_id() to avoid races with destroy

  * Focal update: v5.4.148 upstream stable release (LP: #1946802)
    - rtc: tps65910: Correct driver module alias
    - btrfs: wake up async_delalloc_pages waiters after submit
    - btrfs: reset replace target device to allocation state on close
    - blk-zoned: allow zone management send operations without CAP_SYS_ADMIN
    - blk-zoned: allow BLKREPORTZONE without CAP_SYS_ADMIN
    - PCI/MSI: Skip masking MSI-X on Xen PV
    - powerpc/perf/hv-gpci: Fix counter value parsing
    - xen: fix setting of max_pfn in shared_info
    - include/linux/list.h: add a macro to test if entry is pointing to the head
    - 9p/xen: Fix end of loop tests for list_for_each_entry
    - tools/thermal/tmon: Add cross compiling support
    - pinctrl: stmfx: Fix hazardous u8[] to unsigned long cast
    - pinctrl: ingenic: Fix incorrect pull up/down info
    - soc: qcom: aoss: Fix the out of bound usage of cooling_devs
    - soc: aspeed: lpc-ctrl: Fix boundary check for mmap
    - soc: aspeed: p2a-ctrl: Fix boundary check for mmap
    - arm64: head: avoid over-mapping in map_memory
    - crypto: public_key: fix overflow during implicit conversion
    - block: bfq: fix bfq_set_next_ioprio_data()
    - power: supply: max17042: handle fails of reading status register
    - dm crypt: Avoid percpu_counter spinlock contention in crypt_page_alloc()
    - VMCI: fix NULL pointer dereference when unmapping queue pair
    - media: uvc: don't do DMA on stack
    - media: rc-loopback: return number of emitters rather than error
    - Revert "dmaengine: imx-sdma: refine to load context only once"
    - dmaengine: imx-sdma: remove duplicated sdma_load_context
    - libata: add ATA_HORKAGE_NO_NCQ_TRIM for Samsung 860 and 870 SSDs
    - ARM: 9105/1: atags_to_fdt: don't warn about stack size
    - PCI/portdrv: Enable Bandwidth Notification only if port supports it
    - PCI: Restrict ASMedia ASM1062 SATA Max Payload Size Supported
    - PCI: Return ~0 data on pciconfig_read() CAP_SYS_ADMIN failure
    - PCI: xilinx-nwl: Enable the clock through CCF
    - PCI: aardvark: Fix checking for PIO status
    - PCI: aardvark: Increase polling delay to 1.5s while waiting for PIO response
    - PCI: aardvark: Fix masking and unmasking legacy INTx interrupts
    - HID...