zfcp: fix infinite iteration on ERP ready list
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ubuntu on IBM z Systems |
Fix Released
|
High
|
Canonical Kernel Team | ||
linux (Ubuntu) |
Fix Released
|
High
|
Joseph Salisbury | ||
Bionic |
Fix Released
|
High
|
Joseph Salisbury |
Bug Description
Please backport:
commit fa89adba1941e4f
scsi: zfcp: fix infinite iteration on ERP ready list
zfcp_
rports via zfcp_scsi_
adapter ERP action via zfcp_erp_
processed asynchronously and concurrently.
Blocking of rports is done in a kworker by zfcp_scsi_
calls zfcp_scsi_
zfcp_
and then iterates with list_for_each() over the adapter's ERP ready list
without holding the ERP lock. This opens a race window in which the
current list entry can be moved to another list, causing list_for_each()
to iterate forever on the wrong list, as the erp_ready_head is never
encountered as terminal condition.
Meanwhile the ERP action can be processed in the ERP thread by
zfcp_
lock and then calls zfcp_erp_
from the ready to the running list. zfcp_erp_
move the ERP action using list_move() just during the aforementioned
race window. It then traces a REC RUN "erator1" via zfcp_dbf_rec_run().
zfcp_
held by the infinitely looping kworker, it effectively spins forever.
Example Sequence Diagram:
Process ERP Thread rport_work
---
zfcp_
zfcp_
zfcp_
lock ERP zfcp_scsi_
zfcp_
list_add_tail() on ready !(rport_
wake_up() ERP thread zfcp_scsi_
zfcp_
unlock ERP lock DBF REC
zfcp_erp_wait() lock ERP
| zfcp_erp_
| list_for_each() ready
| list_move() current entry
| ready to running
| zfcp_dbf_rec_run() endless loop over running
| zfcp_dbf_
| lock DBF REC spins forever
Any adapter recovery can trigger this, such as setting the device offline
or reboot.
V4.9 commit 4eeaa4f3f1d6 ("zfcp: close window with unblocked rport
during rport gone") introduced additional tracing of (un)blocking of
rports. It missed that the adapter->erp_lock must be held when calling
zfcp_
This fix uses the approach formerly introduced by commit aa0fec62391c
("[SCSI] zfcp: Fix sparse warning by providing new entry in dbf") that got
later removed by commit ae0904f60fab ("[SCSI] zfcp: Redesign of the debug
tracing for recovery actions.").
Introduce zfcp_dbf_
acquires and releases the adapter->erp_lock for read.
Reported-by: Sebastian Ott <email address hidden>
Signed-off-by: Jens Remus <email address hidden>
Fixes: 4eeaa4f3f1d6 ("zfcp: close window with unblocked rport during rport gone")
Cc: <email address hidden> # 2.6.32+
Reviewed-by: Benjamin Block <email address hidden>
Signed-off-by: Steffen Maier <email address hidden>
Signed-off-by: Martin K. Petersen <email address hidden>
tags: | added: architecture-s39064 bugnameltc-169499 severity-high targetmilestone-inin1804 |
Changed in ubuntu: | |
assignee: | nobody → Skipper Bug Screeners (skipper-screen-team) |
affects: | ubuntu → linux (Ubuntu) |
Changed in ubuntu-z-systems: | |
importance: | Undecided → High |
status: | New → Triaged |
assignee: | nobody → Canonical Kernel Team (canonical-kernel-team) |
Changed in linux (Ubuntu): | |
importance: | Undecided → High |
status: | New → In Progress |
Changed in linux (Ubuntu Bionic): | |
importance: | Undecided → High |
status: | New → In Progress |
assignee: | nobody → Joseph Salisbury (jsalisbury) |
Changed in linux (Ubuntu): | |
assignee: | Skipper Bug Screeners (skipper-screen-team) → Joseph Salisbury (jsalisbury) |
Changed in ubuntu-z-systems: | |
status: | Triaged → In Progress |
Changed in ubuntu-z-systems: | |
status: | In Progress → Fix Committed |
tags: | added: cscc |
built a test kernel with commit fa89adba1941e4f 3b213399b81732a 5c12fd9131. The test kernel can be downloaded from:
http:// kernel. ubuntu. com/~jsalisbury /lp1780067
Can you test this kernel and see if it resolves this bug?
Note about installing test kernels: unsigned .deb packages.
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-
Thanks in advance!