Apr 17 12:18:49 zop03-01 systemd[1]: Started Session 71 of user root. Apr 17 12:18:49 zop03-01 systemd[1]: Starting Session 71 of user root. Apr 17 12:19:37 zop03-01 systemd[1]: Cannot determine UID from slice user-0.slice Apr 17 12:19:37 zop03-01 systemd[1]: Started Session 72 of user root. Apr 17 12:19:37 zop03-01 systemd[1]: Starting Session 72 of user root. Apr 17 12:25:08 zop03-01 kernel: [35488.752833] sd 4:0:0:5: Cancelling outstanding commands. Apr 17 12:25:08 zop03-01 kernel: [35488.778662] sd 4:0:0:5: Successfully cancelled outstanding commands Apr 17 12:25:08 zop03-01 kernel: [35488.778672] sd 4:0:0:5: Aborting outstanding commands Apr 17 12:25:08 zop03-01 kernel: [35488.779299] sd 4:0:0:5: [sdab] Command (2A) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:08 zop03-01 kernel: [35488.779344] sd 4:0:0:5: [sdab] Command (2A) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:09 zop03-01 kernel: [35489.776837] sd 2:0:1:0: Cancelling outstanding commands. Apr 17 12:25:09 zop03-01 kernel: [35489.802898] sd 2:0:1:0: Successfully cancelled outstanding commands Apr 17 12:25:09 zop03-01 kernel: [35489.802908] sd 2:0:1:0: Aborting outstanding commands Apr 17 12:25:09 zop03-01 kernel: [35489.803377] sd 2:0:1:0: [sdf] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:09 zop03-01 kernel: [35489.803646] sd 2:0:1:0: [sdf] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:10 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:25:10 zop03-01 kernel: [35490.680596] device-mapper: multipath: Failing path 65:160. Apr 17 12:25:10 zop03-01 multipathd: checker failed path 65:160 in map mpath6 Apr 17 12:25:10 zop03-01 multipathd: mpath6: remaining active paths: 3 Apr 17 12:25:12 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:25:12 zop03-01 multipathd: checker failed path 8:144 in map mpath2 Apr 17 12:25:12 zop03-01 multipathd: mpath2: remaining active paths: 3 Apr 17 12:25:12 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:25:12 zop03-01 multipathd: checker failed path 65:128 in map mpath9 Apr 17 12:25:12 zop03-01 multipathd: mpath9: remaining active paths: 3 Apr 17 12:25:12 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:25:12 zop03-01 multipathd: checker failed path 65:144 in map mpath5 Apr 17 12:25:12 zop03-01 multipathd: mpath5: remaining active paths: 3 Apr 17 12:25:12 zop03-01 kernel: [35492.684976] device-mapper: multipath: Failing path 8:144. Apr 17 12:25:12 zop03-01 kernel: [35492.686290] device-mapper: multipath: Failing path 65:128. Apr 17 12:25:12 zop03-01 kernel: [35492.686530] device-mapper: multipath: Failing path 65:144. Apr 17 12:25:12 zop03-01 systemd-udevd[18711]: conflicting device node '/dev/mapper/mpath9' found, link to '/dev/dm-23' will not be created Apr 17 12:25:12 zop03-01 systemd-udevd[18712]: conflicting device node '/dev/mapper/mpath5' found, link to '/dev/dm-24' will not be created Apr 17 12:25:12 zop03-01 systemd-udevd[18746]: conflicting device node '/dev/mapper/mpath2-part1' found, link to '/dev/dm-18' will not be created Apr 17 12:25:12 zop03-01 systemd-udevd[18747]: conflicting device node '/dev/mapper/mpath2-part3' found, link to '/dev/dm-20' will not be created Apr 17 12:25:12 zop03-01 systemd-udevd[18709]: conflicting device node '/dev/mapper/mpath2-part2' found, link to '/dev/dm-19' will not be created Apr 17 12:25:13 zop03-01 multipathd: mpath3: sdf - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 8:80 in map mpath3 Apr 17 12:25:13 zop03-01 kernel: [35493.687771] device-mapper: multipath: Failing path 8:80. Apr 17 12:25:13 zop03-01 kernel: [35493.688038] device-mapper: multipath: Failing path 8:96. Apr 17 12:25:13 zop03-01 kernel: [35493.688278] device-mapper: multipath: Failing path 8:112. Apr 17 12:25:13 zop03-01 kernel: [35493.688591] device-mapper: multipath: Failing path 8:128. Apr 17 12:25:13 zop03-01 multipathd: mpath3: remaining active paths: 3 Apr 17 12:25:13 zop03-01 multipathd: mpath0: sdg - directio checker reports path is down Apr 17 12:25:13 zop03-01 kernel: [35493.689029] device-mapper: multipath: Failing path 8:160. Apr 17 12:25:13 zop03-01 kernel: [35493.689365] device-mapper: multipath: Failing path 8:176. Apr 17 12:25:13 zop03-01 kernel: [35493.689619] device-mapper: multipath: Failing path 8:192. Apr 17 12:25:13 zop03-01 kernel: [35493.689870] device-mapper: multipath: Failing path 8:208. Apr 17 12:25:13 zop03-01 kernel: [35493.690121] device-mapper: multipath: Failing path 8:224. Apr 17 12:25:13 zop03-01 kernel: [35493.690307] device-mapper: multipath: Failing path 65:96. Apr 17 12:25:13 zop03-01 kernel: [35493.690479] device-mapper: multipath: Failing path 65:112. Apr 17 12:25:13 zop03-01 kernel: [35493.690660] device-mapper: multipath: Failing path 66:32. Apr 17 12:25:13 zop03-01 kernel: [35493.690842] device-mapper: multipath: Failing path 66:48. Apr 17 12:25:13 zop03-01 kernel: [35493.691053] device-mapper: multipath: Failing path 66:64. Apr 17 12:25:13 zop03-01 kernel: [35493.691283] device-mapper: multipath: Failing path 66:80. Apr 17 12:25:13 zop03-01 kernel: [35493.691464] device-mapper: multipath: Failing path 66:96. Apr 17 12:25:13 zop03-01 multipathd: checker failed path 8:96 in map mpath0 Apr 17 12:25:13 zop03-01 multipathd: mpath0: remaining active paths: 3 Apr 17 12:25:13 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 8:112 in map mpath4 Apr 17 12:25:13 zop03-01 multipathd: mpath4: remaining active paths: 3 Apr 17 12:25:13 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 8:128 in map mpath1 Apr 17 12:25:13 zop03-01 multipathd: mpath1: remaining active paths: 3 Apr 17 12:25:13 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 8:160 in map mpath7 Apr 17 12:25:13 zop03-01 multipathd: mpath7: remaining active paths: 3 Apr 17 12:25:13 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 8:176 in map mpath8 Apr 17 12:25:13 zop03-01 multipathd: mpath8: remaining active paths: 3 Apr 17 12:25:13 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 8:192 in map mpath9 Apr 17 12:25:13 zop03-01 multipathd: mpath9: remaining active paths: 2 Apr 17 12:25:13 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 8:208 in map mpath5 Apr 17 12:25:13 zop03-01 multipathd: mpath5: remaining active paths: 2 Apr 17 12:25:13 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 8:224 in map mpath6 Apr 17 12:25:13 zop03-01 multipathd: mpath6: remaining active paths: 2 Apr 17 12:25:13 zop03-01 multipathd: mpath7: sdw - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 65:96 in map mpath7 Apr 17 12:25:13 zop03-01 multipathd: mpath7: remaining active paths: 2 Apr 17 12:25:13 zop03-01 multipathd: mpath8: sdx - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 65:112 in map mpath8 Apr 17 12:25:13 zop03-01 multipathd: mpath8: remaining active paths: 2 Apr 17 12:25:13 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 66:32 in map mpath3 Apr 17 12:25:13 zop03-01 multipathd: mpath3: remaining active paths: 2 Apr 17 12:25:13 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 66:48 in map mpath0 Apr 17 12:25:13 zop03-01 multipathd: mpath0: remaining active paths: 2 Apr 17 12:25:13 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 66:64 in map mpath4 Apr 17 12:25:13 zop03-01 multipathd: mpath4: remaining active paths: 2 Apr 17 12:25:13 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 66:80 in map mpath1 Apr 17 12:25:13 zop03-01 multipathd: mpath1: remaining active paths: 2 Apr 17 12:25:13 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:25:13 zop03-01 multipathd: checker failed path 66:96 in map mpath2 Apr 17 12:25:13 zop03-01 multipathd: mpath2: remaining active paths: 2 Apr 17 12:25:13 zop03-01 systemd-udevd[18746]: conflicting device node '/dev/mapper/mpath0' found, link to '/dev/dm-21' will not be created Apr 17 12:25:13 zop03-01 systemd-udevd[18747]: conflicting device node '/dev/mapper/mpath4' found, link to '/dev/dm-22' will not be created Apr 17 12:25:13 zop03-01 systemd-udevd[18854]: conflicting device node '/dev/mapper/mpath3-part1' found, link to '/dev/dm-14' will not be created Apr 17 12:25:13 zop03-01 systemd-udevd[18864]: conflicting device node '/dev/mapper/mpath1-part1' found, link to '/dev/dm-6' will not be created Apr 17 12:25:13 zop03-01 systemd-udevd[18862]: conflicting device node '/dev/mapper/mpath3-part2' found, link to '/dev/dm-15' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18869]: conflicting device node '/dev/mapper/mpath0-part1' found, link to '/dev/dm-26' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18876]: conflicting device node '/dev/mapper/mpath1-part2' found, link to '/dev/dm-7' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18878]: conflicting device node '/dev/mapper/mpath0-part2' found, link to '/dev/dm-28' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18796]: conflicting device node '/dev/mapper/mpath4-part1' found, link to '/dev/dm-25' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18919]: conflicting device node '/dev/mapper/mpath1-part3' found, link to '/dev/dm-8' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18950]: conflicting device node '/dev/mapper/mpath4-part3' found, link to '/dev/dm-29' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18916]: conflicting device node '/dev/mapper/mpath3-part3' found, link to '/dev/dm-16' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18929]: conflicting device node '/dev/mapper/mpath2-part2' found, link to '/dev/dm-19' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18940]: conflicting device node '/dev/mapper/mpath8-part1' found, link to '/dev/dm-2' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18947]: conflicting device node '/dev/mapper/mpath7-part1' found, link to '/dev/dm-11' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18925]: conflicting device node '/dev/mapper/mpath2-part1' found, link to '/dev/dm-18' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18934]: conflicting device node '/dev/mapper/mpath2-part3' found, link to '/dev/dm-20' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18965]: conflicting device node '/dev/mapper/mpath7-part2' found, link to '/dev/dm-12' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18937]: conflicting device node '/dev/mapper/mpath0-part3' found, link to '/dev/dm-31' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18944]: conflicting device node '/dev/mapper/mpath4-part2' found, link to '/dev/dm-27' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18952]: conflicting device node '/dev/mapper/mpath8-part3' found, link to '/dev/dm-4' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18977]: conflicting device node '/dev/mapper/mpath7-part3' found, link to '/dev/dm-13' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18949]: conflicting device node '/dev/mapper/mpath8-part2' found, link to '/dev/dm-3' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18854]: conflicting device node '/dev/mapper/mpath1-part1' found, link to '/dev/dm-6' will not be created Apr 17 12:25:14 zop03-01 kernel: [35494.528314] sd 3:0:0:0: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.528473] sd 5:0:0:0: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.529306] sd 3:0:0:0: [sdk] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.529380] sd 3:0:0:0: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 kernel: [35494.529431] sd 3:0:0:1: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.538252] sd 3:0:0:1: [sdl] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.538306] sd 5:0:0:0: [sdai] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.538345] sd 3:0:0:1: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 kernel: [35494.538382] sd 5:0:0:0: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 kernel: [35494.538392] sd 3:0:0:2: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.538439] sd 5:0:0:1: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.548175] sd 5:0:0:1: [sdaj] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.548179] sd 3:0:0:2: [sdm] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.548216] sd 3:0:0:2: [sdm] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 systemd-udevd[18869]: conflicting device node '/dev/mapper/mpath3-part2' found, link to '/dev/dm-15' will not be created Apr 17 12:25:14 zop03-01 kernel: [35494.548287] sd 3:0:0:2: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 kernel: [35494.548303] sd 5:0:0:1: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 kernel: [35494.548343] sd 5:0:0:2: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.548353] sd 3:0:0:3: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.549267] sd 5:0:0:2: [sdak] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.549288] sd 5:0:0:2: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 kernel: [35494.549308] sd 5:0:0:3: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.549340] sd 3:0:0:3: [sdn] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.549361] sd 3:0:0:3: [sdn] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.549396] sd 3:0:0:3: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 kernel: [35494.549420] sd 3:0:0:4: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.558355] sd 5:0:0:3: [sdal] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.558360] sd 3:0:0:4: [sdo] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.558367] sd 3:0:0:4: [sdo] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.558436] sd 3:0:0:4: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 systemd-udevd[18862]: conflicting device node '/dev/mapper/mpath1-part2' found, link to '/dev/dm-7' will not be created Apr 17 12:25:14 zop03-01 kernel: [35494.558595] sd 5:0:0:3: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 kernel: [35494.558618] sd 5:0:0:4: Cancelling outstanding commands. Apr 17 12:25:14 zop03-01 kernel: [35494.568328] sd 5:0:0:4: [sdam] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:14 zop03-01 kernel: [35494.568401] sd 5:0:0:4: Successfully cancelled outstanding commands Apr 17 12:25:14 zop03-01 systemd-udevd[18864]: conflicting device node '/dev/mapper/mpath3-part1' found, link to '/dev/dm-14' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18876]: conflicting device node '/dev/mapper/mpath0-part1' found, link to '/dev/dm-26' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18786]: conflicting device node '/dev/mapper/mpath0-part2' found, link to '/dev/dm-28' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18709]: conflicting device node '/dev/mapper/mpath3-part3' found, link to '/dev/dm-16' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18916]: conflicting device node '/dev/mapper/mpath1-part3' found, link to '/dev/dm-8' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18919]: conflicting device node '/dev/mapper/mpath0-part3' found, link to '/dev/dm-31' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18746]: conflicting device node '/dev/mapper/mpath0' found, link to '/dev/dm-21' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18746]: conflicting device node '/dev/mapper/mpath4-part3' found, link to '/dev/dm-29' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18878]: conflicting device node '/dev/mapper/mpath7-part1' found, link to '/dev/dm-11' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18929]: conflicting device node '/dev/mapper/mpath8-part1' found, link to '/dev/dm-2' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18925]: conflicting device node '/dev/mapper/mpath7-part2' found, link to '/dev/dm-12' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18747]: conflicting device node '/dev/mapper/mpath4' found, link to '/dev/dm-22' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18747]: conflicting device node '/dev/mapper/mpath4-part2' found, link to '/dev/dm-27' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18934]: conflicting device node '/dev/mapper/mpath8-part3' found, link to '/dev/dm-4' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18788]: conflicting device node '/dev/mapper/mpath7-part3' found, link to '/dev/dm-13' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18789]: conflicting device node '/dev/mapper/mpath8-part2' found, link to '/dev/dm-3' will not be created Apr 17 12:25:15 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:25:17 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:25:17 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:25:17 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath3: sdf - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath0: sdg - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath7: sdw - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath8: sdx - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:18 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:25:20 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:25:22 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:25:22 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:25:22 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath3: sdf - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath0: sdg - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath7: sdw - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath8: sdx - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:23 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:25:25 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:25:27 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:25:27 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:25:27 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath3: sdf - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath0: sdg - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath7: sdw - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath8: sdx - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:28 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:25:30 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:25:32 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:25:32 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:25:32 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath3: sdf - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath0: sdg - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath7: sdw - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath8: sdx - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:33 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:25:35 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:25:37 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:25:37 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:25:37 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath3: sdf - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath0: sdg - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath7: sdw - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath8: sdx - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:38 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:25:40 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:25:42 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:25:42 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:25:42 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath3: sdf - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath0: sdg - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath7: sdw - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath8: sdx - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:43 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:25:44 zop03-01 kernel: [35524.840787] sd 5:0:0:1: Cancelling outstanding commands. Apr 17 12:25:44 zop03-01 kernel: [35524.841752] sd 5:0:0:1: [sdaj] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:44 zop03-01 kernel: [35524.841792] sd 5:0:0:1: Successfully cancelled outstanding commands Apr 17 12:25:44 zop03-01 kernel: [35524.841805] sd 5:0:0:2: Cancelling outstanding commands. Apr 17 12:25:44 zop03-01 kernel: [35524.842736] sd 5:0:0:2: [sdak] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:44 zop03-01 kernel: [35524.842751] sd 5:0:0:2: Successfully cancelled outstanding commands Apr 17 12:25:44 zop03-01 kernel: [35524.842759] sd 5:0:0:0: Cancelling outstanding commands. Apr 17 12:25:44 zop03-01 kernel: [35524.843690] sd 5:0:0:0: [sdai] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:44 zop03-01 kernel: [35524.843703] sd 5:0:0:0: Successfully cancelled outstanding commands Apr 17 12:25:44 zop03-01 kernel: [35524.843710] sd 5:0:0:3: Cancelling outstanding commands. Apr 17 12:25:44 zop03-01 kernel: [35524.844659] sd 5:0:0:3: [sdal] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:44 zop03-01 kernel: [35524.844672] sd 5:0:0:3: Successfully cancelled outstanding commands Apr 17 12:25:44 zop03-01 kernel: [35524.844679] sd 5:0:0:4: Cancelling outstanding commands. Apr 17 12:25:44 zop03-01 kernel: [35524.845630] sd 5:0:0:4: [sdam] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:44 zop03-01 kernel: [35524.845644] sd 5:0:0:4: Successfully cancelled outstanding commands Apr 17 12:25:45 zop03-01 kernel: [35525.480762] sd 3:0:0:0: Cancelling outstanding commands. Apr 17 12:25:45 zop03-01 kernel: [35525.481757] sd 3:0:0:0: [sdk] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:45 zop03-01 kernel: [35525.481787] sd 3:0:0:0: Successfully cancelled outstanding commands Apr 17 12:25:45 zop03-01 kernel: [35525.481809] sd 3:0:0:3: Cancelling outstanding commands. Apr 17 12:25:45 zop03-01 kernel: [35525.482930] sd 3:0:0:3: [sdn] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:45 zop03-01 kernel: [35525.482944] sd 3:0:0:3: [sdn] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:45 zop03-01 kernel: [35525.482956] sd 3:0:0:3: Successfully cancelled outstanding commands Apr 17 12:25:45 zop03-01 kernel: [35525.482965] sd 3:0:0:4: Cancelling outstanding commands. Apr 17 12:25:45 zop03-01 kernel: [35525.483939] sd 3:0:0:4: [sdo] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:45 zop03-01 kernel: [35525.483954] sd 3:0:0:4: [sdo] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:45 zop03-01 kernel: [35525.483965] sd 3:0:0:4: Successfully cancelled outstanding commands Apr 17 12:25:45 zop03-01 kernel: [35525.483972] sd 3:0:0:1: Cancelling outstanding commands. Apr 17 12:25:45 zop03-01 kernel: [35525.484928] sd 3:0:0:1: [sdl] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:45 zop03-01 kernel: [35525.484950] sd 3:0:0:1: Successfully cancelled outstanding commands Apr 17 12:25:45 zop03-01 kernel: [35525.484959] sd 3:0:0:2: Cancelling outstanding commands. Apr 17 12:25:45 zop03-01 kernel: [35525.485933] sd 3:0:0:2: [sdm] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:45 zop03-01 kernel: [35525.485947] sd 3:0:0:2: [sdm] Command (28) failed: transaction cancelled (200:600) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:45 zop03-01 kernel: [35525.485959] sd 3:0:0:2: Successfully cancelled outstanding commands Apr 17 12:25:45 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:25:47 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:25:47 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:25:47 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath3: sdf - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath0: sdg - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath7: sdw - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath8: sdx - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:48 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:25:48 zop03-01 kernel: [35528.808749] ibmvfc 30000005: Timed out waiting for aborted commands Apr 17 12:25:48 zop03-01 kernel: [35528.808769] sd 4:0:0:5: Cancel failed, resetting host Apr 17 12:25:49 zop03-01 kernel: [35529.833079] ibmvfc 30000003: Timed out waiting for aborted commands Apr 17 12:25:49 zop03-01 kernel: [35529.833094] sd 2:0:1:0: Cancel failed, resetting host Apr 17 12:25:50 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:25:52 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:25:52 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:25:52 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:25:53 zop03-01 kernel: [35533.203559] sd 4:0:0:3: [sdz] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:53 zop03-01 kernel: [35533.229623] sd 4:0:0:4: [sdaa] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:53 zop03-01 kernel: [35533.294725] sd 4:0:0:0: [sdw] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:53 zop03-01 kernel: [35533.312809] sd 4:0:0:2: [sdy] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:53 zop03-01 kernel: [35533.313426] sd 4:0:0:1: [sdx] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:53 zop03-01 kernel: [35533.330079] ibmvfc 30000005: Host partition: zopv2, device: vfchost8 U78CB.001.WZS047U-P1-C3-T2 U8247.22L.2122A6A-V3-C22 max sectors 2048 Apr 17 12:25:53 zop03-01 multipathd: mpath3: sdf - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath0: sdg - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:53 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:25:54 zop03-01 kernel: [35534.673365] sd 2:0:1:2: [sdh] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:54 zop03-01 kernel: [35534.738217] sd 2:0:1:2: [sdh] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:54 zop03-01 kernel: [35534.768489] sd 2:0:1:4: [sdj] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:54 zop03-01 kernel: [35534.782384] sd 2:0:1:1: [sdg] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:54 zop03-01 kernel: [35534.782713] sd 2:0:1:3: [sdi] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:54 zop03-01 kernel: [35534.782955] sd 2:0:1:1: [sdg] Command (28) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:25:54 zop03-01 kernel: [35534.805058] ibmvfc 30000003: Host partition: zopv1, device: vfchost8 U78CB.001.WZS047U-P1-C7-T2 U8247.22L.2122A6A-V2-C22 max sectors 2048 Apr 17 12:25:58 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:25:58 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:25:58 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:25:58 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:25:58 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:25:58 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:25:58 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:25:58 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:25:58 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:25:58 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:26:03 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath7: sdk - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath8: sdl - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath9: sdm - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath5: sdn - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath6: sdo - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath3: sdai - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath0: sdaj - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath4: sdak - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath1: sdal - directio checker reports path is down Apr 17 12:26:08 zop03-01 multipathd: mpath2: sdam - directio checker reports path is down Apr 17 12:26:09 zop03-01 kernel: [35549.289735] ibmvfc 30000006: 298400: ADISC failed: transaction cancelled (200:600) unknown fc type (0) no additional explanation (0) rc=0xF7 Apr 17 12:26:09 zop03-01 kernel: [35549.289895] ibmvfc 30000004: 298400: ADISC failed: transaction cancelled (200:600) unknown fc type (0) no additional explanation (0) rc=0xF7 Apr 17 12:26:09 zop03-01 kernel: [35549.291640] ibmvfc 30000006: 298400: Implicit Logout failed: rc=0xF7 Apr 17 12:26:09 zop03-01 kernel: [35549.292516] ibmvfc 30000004: 298400: Implicit Logout failed: rc=0xF7 Apr 17 12:26:11 zop03-01 systemd-udevd[13255]: worker [18698] /devices/virtual/block/dm-5 is taking a long time Apr 17 12:26:14 zop03-01 systemd-udevd[13255]: worker [18711] /devices/virtual/block/dm-23 is taking a long time Apr 17 12:26:14 zop03-01 systemd-udevd[13255]: worker [18712] /devices/virtual/block/dm-24 is taking a long time Apr 17 12:26:23 zop03-01 kernel: [35563.558316] ibmvfc 30000004: 298400: Port Login failed: transaction cancelled (2200:600) unknown fc type (0) no additional explanation (0) rc=0xF7 Apr 17 12:26:23 zop03-01 kernel: [35563.558341] sd 3:0:0:0: [sdk] Command (00) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:26:24 zop03-01 kernel: [35564.144878] ibmvfc 30000006: 298400: Port Login failed: transaction cancelled (2200:600) unknown fc type (0) no additional explanation (0) rc=0xF7 Apr 17 12:26:24 zop03-01 kernel: [35564.144896] sd 5:0:0:1: [sdaj] Command (00) failed: transaction cancelled implicit (200:700) flags: 0 fcp_rsp: 0, resid=0, scsi_status: 0 Apr 17 12:27:16 zop03-01 kernel: [35616.841436] ibmvfc 30000005: 298400: Port Login failed: unable to establish (2100:100) unknown fc type (0) no additional explanation (0) rc=0xF7 Apr 17 12:27:18 zop03-01 kernel: [35618.852794] ibmvfc 30000003: 298400: Port Login failed: unable to establish (2100:100) unknown fc type (0) no additional explanation (0) rc=0xF7 Apr 17 12:27:34 zop03-01 systemd[1]: Cannot determine UID from slice user-0.slice Apr 17 12:27:34 zop03-01 systemd[1]: Started Session 73 of user root. Apr 17 12:27:34 zop03-01 systemd[1]: Starting Session 73 of user root. Apr 17 12:27:40 zop03-01 kernel: [35640.184808] INFO: task jbd2/sdab2-8:13145 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.184829] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.184833] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.184838] jbd2/sdab2-8 D 0000000000000000 0 13145 2 0x00000800 Apr 17 12:27:40 zop03-01 kernel: [35640.184842] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.184847] [c0000003b9683530] [c000000000000000] _stext+0x0/0x30 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.184851] [c0000003b9683700] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.184855] [c0000003b9683750] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.184858] [c0000003b9683980] [c000000000a349a4] io_schedule+0xb4/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.184860] [c0000003b96839b0] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.184862] [c0000003b96839d0] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.184865] [c0000003b9683a20] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.184867] [c0000003b9683aa0] [c000000000307830] __wait_on_buffer+0x50/0x70 Apr 17 12:27:40 zop03-01 kernel: [35640.184870] [c0000003b9683ad0] [c0000000003d6e88] jbd2_journal_commit_transaction+0x10a8/0x1ec0 Apr 17 12:27:40 zop03-01 kernel: [35640.184873] [c0000003b9683cf0] [c0000000003def3c] kjournald2+0xec/0x310 Apr 17 12:27:40 zop03-01 kernel: [35640.184875] [c0000003b9683d80] [c0000000000dae90] kthread+0x110/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.184878] [c0000003b9683e30] [c000000000009560] ret_from_kernel_thread+0x5c/0x7c Apr 17 12:27:40 zop03-01 kernel: [35640.184904] INFO: task kworker/u320:1:18262 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.184910] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.184913] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.184918] kworker/u320:1 D 0000000000000000 0 18262 2 0x00000800 Apr 17 12:27:40 zop03-01 kernel: [35640.184923] Workqueue: writeback bdi_writeback_workfn (flush-65:176) Apr 17 12:27:40 zop03-01 kernel: [35640.184925] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.184935] [c0000003b40f6d60] [c0000003b40f6da0] 0xc0000003b40f6da0 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.184938] [c0000003b40f6f30] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.184941] [c0000003b40f6f80] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.184943] --- interrupt: e40 at 0xc0000003b40f7130 Apr 17 12:27:40 zop03-01 kernel: [35640.184943] LR = (null) Apr 17 12:27:40 zop03-01 kernel: [35640.184946] [c0000003b40f71b0] [c000000000a349a4] io_schedule+0xb4/0x130 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.184948] [c0000003b40f71e0] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.184950] [c0000003b40f7200] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.184953] [c0000003b40f7250] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.184955] [c0000003b40f72d0] [c0000000003d3d30] do_get_write_access+0x3b0/0x670 Apr 17 12:27:40 zop03-01 kernel: [35640.184957] [c0000003b40f73b0] [c0000000003d403c] jbd2_journal_get_write_access+0x4c/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.184960] [c0000003b40f73e0] [c0000000003b347c] __ext4_journal_get_write_access+0x8c/0xe0 Apr 17 12:27:40 zop03-01 kernel: [35640.184962] [c0000003b40f7420] [c0000000003bcd40] ext4_mb_mark_diskspace_used+0x150/0x570 Apr 17 12:27:40 zop03-01 kernel: [35640.184964] [c0000003b40f74e0] [c0000000003be9d4] ext4_mb_new_blocks+0x444/0x620 Apr 17 12:27:40 zop03-01 kernel: [35640.184967] [c0000003b40f7560] [c0000000003afc54] ext4_ext_map_blocks+0x664/0x1300 Apr 17 12:27:40 zop03-01 kernel: [35640.184969] [c0000003b40f7690] [c000000000377878] ext4_map_blocks+0x168/0x700 Apr 17 12:27:40 zop03-01 kernel: [35640.184971] [c0000003b40f7760] [c00000000037b8e4] ext4_writepages+0x6f4/0xd20 Apr 17 12:27:40 zop03-01 kernel: [35640.184974] [c0000003b40f78e0] [c000000000224180] do_writepages+0x60/0xc0 Apr 17 12:27:40 zop03-01 kernel: [35640.184977] [c0000003b40f7910] [c0000000002fd108] __writeback_single_inode+0x68/0x370 Apr 17 12:27:40 zop03-01 kernel: [35640.184979] [c0000003b40f7960] [c0000000002fea58] writeback_sb_inodes+0x2c8/0x4f0 Apr 17 12:27:40 zop03-01 kernel: [35640.184982] [c0000003b40f7a30] [c0000000002fed64] __writeback_inodes_wb+0xe4/0x150 Apr 17 12:27:40 zop03-01 kernel: [35640.184984] [c0000003b40f7a90] [c0000000002ff0cc] wb_writeback+0x2fc/0x400 Apr 17 12:27:40 zop03-01 kernel: [35640.184986] [c0000003b40f7b30] [c0000000002ff7a0] bdi_writeback_workfn+0x310/0x580 Apr 17 12:27:40 zop03-01 kernel: [35640.184989] [c0000003b40f7c50] [c0000000000d31b4] process_one_work+0x1a4/0x4a0 Apr 17 12:27:40 zop03-01 kernel: [35640.184992] [c0000003b40f7ce0] [c0000000000d3aa0] worker_thread+0x190/0x600 Apr 17 12:27:40 zop03-01 kernel: [35640.184994] [c0000003b40f7d80] [c0000000000dae90] kthread+0x110/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.184997] [c0000003b40f7e30] [c000000000009560] ret_from_kernel_thread+0x5c/0x7c Apr 17 12:27:40 zop03-01 kernel: [35640.185000] INFO: task latest_blast:18543 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.185005] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.185008] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.185013] latest_blast D 00003fff9e03388c 0 18543 18377 0x00040000 Apr 17 12:27:40 zop03-01 kernel: [35640.185015] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.185017] [c0000003ba3fb200] [c0000003ba3fb270] 0xc0000003ba3fb270 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.185020] [c0000003ba3fb3d0] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.185022] [c0000003ba3fb420] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185024] [c0000003ba3fb650] [c000000000a349a4] io_schedule+0xb4/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.185027] [c0000003ba3fb680] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185029] [c0000003ba3fb6a0] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.185031] [c0000003ba3fb6f0] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.185033] [c0000003ba3fb770] [c0000000003d3d30] do_get_write_access+0x3b0/0x670 Apr 17 12:27:40 zop03-01 kernel: [35640.185035] [c0000003ba3fb850] [c0000000003d403c] jbd2_journal_get_write_access+0x4c/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185037] [c0000003ba3fb880] [c0000000003b347c] __ext4_journal_get_write_access+0x8c/0xe0 Apr 17 12:27:40 zop03-01 kernel: [35640.185040] [c0000003ba3fb8c0] [c000000000372ba8] __ext4_new_inode+0x5d8/0x16d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185042] [c0000003ba3fba10] [c000000000387524] ext4_create+0x154/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185044] [c0000003ba3fbae0] [c0000000002d0798] vfs_create+0x178/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185046] [c0000003ba3fbb30] [c0000000002d175c] do_last+0xf2c/0x1000 Apr 17 12:27:40 zop03-01 kernel: [35640.185048] [c0000003ba3fbc00] [c0000000002d45bc] path_openat+0xdc/0x810 Apr 17 12:27:40 zop03-01 kernel: [35640.185050] [c0000003ba3fbcd0] [c0000000002d5c48] do_filp_open+0x58/0xf0 Apr 17 12:27:40 zop03-01 kernel: [35640.185053] [c0000003ba3fbdb0] [c0000000002bcbf8] do_sys_open+0x1c8/0x390 Apr 17 12:27:40 zop03-01 kernel: [35640.185055] [c0000003ba3fbe30] [c000000000009258] syscall_exit+0x0/0x98 Apr 17 12:27:40 zop03-01 kernel: [35640.185057] INFO: task latest_blast:18544 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.185062] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.185065] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.185070] latest_blast D 00003fff9e032ce4 0 18544 18377 0x00040000 Apr 17 12:27:40 zop03-01 kernel: [35640.185072] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.185075] [c0000003b799f2a0] [c000000000eba880] runqueues+0x0/0xa80 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.185078] [c0000003b799f470] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.185080] [c0000003b799f4c0] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185082] [c0000003b799f6f0] [c000000000a349a4] io_schedule+0xb4/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.185084] [c0000003b799f720] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185087] [c0000003b799f740] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.185089] [c0000003b799f790] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.185091] [c0000003b799f810] [c0000000003d3d30] do_get_write_access+0x3b0/0x670 Apr 17 12:27:40 zop03-01 kernel: [35640.185093] [c0000003b799f8f0] [c0000000003d403c] jbd2_journal_get_write_access+0x4c/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185095] [c0000003b799f920] [c0000000003b347c] __ext4_journal_get_write_access+0x8c/0xe0 Apr 17 12:27:40 zop03-01 kernel: [35640.185097] [c0000003b799f960] [c00000000037af5c] ext4_reserve_inode_write+0xcc/0x100 Apr 17 12:27:40 zop03-01 kernel: [35640.185099] [c0000003b799f9b0] [c00000000037aff0] ext4_mark_inode_dirty+0x60/0x260 Apr 17 12:27:40 zop03-01 kernel: [35640.185101] [c0000003b799fa60] [c00000000037ecec] ext4_dirty_inode+0x5c/0x90 Apr 17 12:27:40 zop03-01 kernel: [35640.185104] [c0000003b799fa90] [c0000000002fdabc] __mark_inode_dirty+0x6c/0x340 Apr 17 12:27:40 zop03-01 kernel: [35640.185106] [c0000003b799fb10] [c0000000002e54ec] update_time+0xdc/0x150 Apr 17 12:27:40 zop03-01 kernel: [35640.185108] [c0000003b799fb50] [c0000000002e5828] file_update_time+0xd8/0x140 Apr 17 12:27:40 zop03-01 kernel: [35640.185111] [c0000003b799fba0] [c000000000214758] __generic_file_write_iter+0x278/0x430 Apr 17 12:27:40 zop03-01 kernel: [35640.185113] [c0000003b799fc00] [c00000000036ffe8] ext4_file_write_iter+0x118/0x480 Apr 17 12:27:40 zop03-01 kernel: [35640.185115] [c0000003b799fcc0] [c0000000002bd93c] new_sync_write+0xac/0x120 Apr 17 12:27:40 zop03-01 kernel: [35640.185117] [c0000003b799fd90] [c0000000002be4fc] vfs_write+0xdc/0x260 Apr 17 12:27:40 zop03-01 kernel: [35640.185119] [c0000003b799fde0] [c0000000002bf37c] SyS_write+0x6c/0x110 Apr 17 12:27:40 zop03-01 kernel: [35640.185121] [c0000003b799fe30] [c000000000009258] syscall_exit+0x0/0x98 Apr 17 12:27:40 zop03-01 kernel: [35640.185123] INFO: task latest_blast:18545 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.185128] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.185131] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.185136] latest_blast D 00003fff9e032ce4 0 18545 18377 0x00040000 Apr 17 12:27:40 zop03-01 kernel: [35640.185138] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.185140] [c0000003b798b2a0] [c0000003b798b300] 0xc0000003b798b300 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.185143] [c0000003b798b470] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.185145] [c0000003b798b4c0] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185147] [c0000003b798b6f0] [c000000000a349a4] io_schedule+0xb4/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.185149] [c0000003b798b720] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185152] [c0000003b798b740] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.185154] [c0000003b798b790] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.185156] [c0000003b798b810] [c0000000003d3d30] do_get_write_access+0x3b0/0x670 Apr 17 12:27:40 zop03-01 kernel: [35640.185158] [c0000003b798b8f0] [c0000000003d403c] jbd2_journal_get_write_access+0x4c/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185160] [c0000003b798b920] [c0000000003b347c] __ext4_journal_get_write_access+0x8c/0xe0 Apr 17 12:27:40 zop03-01 kernel: [35640.185162] [c0000003b798b960] [c00000000037af5c] ext4_reserve_inode_write+0xcc/0x100 Apr 17 12:27:40 zop03-01 kernel: [35640.185164] [c0000003b798b9b0] [c00000000037aff0] ext4_mark_inode_dirty+0x60/0x260 Apr 17 12:27:40 zop03-01 kernel: [35640.185166] [c0000003b798ba60] [c00000000037ecec] ext4_dirty_inode+0x5c/0x90 Apr 17 12:27:40 zop03-01 kernel: [35640.185169] [c0000003b798ba90] [c0000000002fdabc] __mark_inode_dirty+0x6c/0x340 Apr 17 12:27:40 zop03-01 kernel: [35640.185170] [c0000003b798bb10] [c0000000002e54ec] update_time+0xdc/0x150 Apr 17 12:27:40 zop03-01 kernel: [35640.185172] [c0000003b798bb50] [c0000000002e5828] file_update_time+0xd8/0x140 Apr 17 12:27:40 zop03-01 kernel: [35640.185175] [c0000003b798bba0] [c000000000214758] __generic_file_write_iter+0x278/0x430 Apr 17 12:27:40 zop03-01 kernel: [35640.185177] [c0000003b798bc00] [c00000000036ffe8] ext4_file_write_iter+0x118/0x480 Apr 17 12:27:40 zop03-01 kernel: [35640.185179] [c0000003b798bcc0] [c0000000002bd93c] new_sync_write+0xac/0x120 Apr 17 12:27:40 zop03-01 kernel: [35640.185181] [c0000003b798bd90] [c0000000002be4fc] vfs_write+0xdc/0x260 Apr 17 12:27:40 zop03-01 kernel: [35640.185183] [c0000003b798bde0] [c0000000002bf37c] SyS_write+0x6c/0x110 Apr 17 12:27:40 zop03-01 kernel: [35640.185185] [c0000003b798be30] [c000000000009258] syscall_exit+0x0/0x98 Apr 17 12:27:40 zop03-01 kernel: [35640.185187] INFO: task latest_blast:18546 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.185192] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.185195] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.185199] latest_blast D 00003fff9e03388c 0 18546 18377 0x00040000 Apr 17 12:27:40 zop03-01 kernel: [35640.185202] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.185204] [c0000003b79ab200] [c0000003b79ab250] 0xc0000003b79ab250 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.185206] [c0000003b79ab3d0] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.185209] [c0000003b79ab420] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185211] [c0000003b79ab650] [c000000000a349a4] io_schedule+0xb4/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.185213] [c0000003b79ab680] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185215] [c0000003b79ab6a0] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.185218] [c0000003b79ab6f0] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.185220] [c0000003b79ab770] [c0000000003d3d30] do_get_write_access+0x3b0/0x670 Apr 17 12:27:40 zop03-01 kernel: [35640.185222] [c0000003b79ab850] [c0000000003d403c] jbd2_journal_get_write_access+0x4c/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185224] [c0000003b79ab880] [c0000000003b347c] __ext4_journal_get_write_access+0x8c/0xe0 Apr 17 12:27:40 zop03-01 kernel: [35640.185226] [c0000003b79ab8c0] [c000000000372ba8] __ext4_new_inode+0x5d8/0x16d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185228] [c0000003b79aba10] [c000000000387524] ext4_create+0x154/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185230] [c0000003b79abae0] [c0000000002d0798] vfs_create+0x178/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185232] [c0000003b79abb30] [c0000000002d175c] do_last+0xf2c/0x1000 Apr 17 12:27:40 zop03-01 kernel: [35640.185234] [c0000003b79abc00] [c0000000002d45bc] path_openat+0xdc/0x810 Apr 17 12:27:40 zop03-01 kernel: [35640.185236] [c0000003b79abcd0] [c0000000002d5c48] do_filp_open+0x58/0xf0 Apr 17 12:27:40 zop03-01 kernel: [35640.185238] [c0000003b79abdb0] [c0000000002bcbf8] do_sys_open+0x1c8/0x390 Apr 17 12:27:40 zop03-01 kernel: [35640.185241] [c0000003b79abe30] [c000000000009258] syscall_exit+0x0/0x98 Apr 17 12:27:40 zop03-01 kernel: [35640.185242] INFO: task latest_blast:18547 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.185247] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.185250] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.185255] latest_blast D 00003fff9e03388c 0 18547 18377 0x00040000 Apr 17 12:27:40 zop03-01 kernel: [35640.185257] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.185259] [c0000003a64bb200] [c0000003a64bb250] 0xc0000003a64bb250 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.185262] [c0000003a64bb3d0] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.185264] [c0000003a64bb420] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185266] [c0000003a64bb650] [c000000000a349a4] io_schedule+0xb4/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.185269] [c0000003a64bb680] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185271] [c0000003a64bb6a0] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.185273] [c0000003a64bb6f0] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.185275] [c0000003a64bb770] [c0000000003d3d30] do_get_write_access+0x3b0/0x670 Apr 17 12:27:40 zop03-01 kernel: [35640.185277] [c0000003a64bb850] [c0000000003d403c] jbd2_journal_get_write_access+0x4c/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185279] [c0000003a64bb880] [c0000000003b347c] __ext4_journal_get_write_access+0x8c/0xe0 Apr 17 12:27:40 zop03-01 kernel: [35640.185281] [c0000003a64bb8c0] [c000000000372ba8] __ext4_new_inode+0x5d8/0x16d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185283] [c0000003a64bba10] [c000000000387524] ext4_create+0x154/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185285] [c0000003a64bbae0] [c0000000002d0798] vfs_create+0x178/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185287] [c0000003a64bbb30] [c0000000002d175c] do_last+0xf2c/0x1000 Apr 17 12:27:40 zop03-01 kernel: [35640.185289] [c0000003a64bbc00] [c0000000002d45bc] path_openat+0xdc/0x810 Apr 17 12:27:40 zop03-01 kernel: [35640.185291] [c0000003a64bbcd0] [c0000000002d5c48] do_filp_open+0x58/0xf0 Apr 17 12:27:40 zop03-01 kernel: [35640.185293] [c0000003a64bbdb0] [c0000000002bcbf8] do_sys_open+0x1c8/0x390 Apr 17 12:27:40 zop03-01 kernel: [35640.185296] [c0000003a64bbe30] [c000000000009258] syscall_exit+0x0/0x98 Apr 17 12:27:40 zop03-01 kernel: [35640.185297] INFO: task latest_blast:18548 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.185302] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.185305] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.185310] latest_blast D 00003fff9e03388c 0 18548 18377 0x00040000 Apr 17 12:27:40 zop03-01 kernel: [35640.185312] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.185314] [c0000003a64b33d0] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.185317] [c0000003a64b3420] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185319] [c0000003a64b3650] [c000000000a349a4] io_schedule+0xb4/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.185321] [c0000003a64b3680] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185323] [c0000003a64b36a0] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.185326] [c0000003a64b36f0] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.185328] [c0000003a64b3770] [c0000000003d3d30] do_get_write_access+0x3b0/0x670 Apr 17 12:27:40 zop03-01 kernel: [35640.185330] [c0000003a64b3850] [c0000000003d403c] jbd2_journal_get_write_access+0x4c/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185332] [c0000003a64b3880] [c0000000003b347c] __ext4_journal_get_write_access+0x8c/0xe0 Apr 17 12:27:40 zop03-01 kernel: [35640.185334] [c0000003a64b38c0] [c000000000372ba8] __ext4_new_inode+0x5d8/0x16d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185336] [c0000003a64b3a10] [c000000000387524] ext4_create+0x154/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185338] [c0000003a64b3ae0] [c0000000002d0798] vfs_create+0x178/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185340] [c0000003a64b3b30] [c0000000002d175c] do_last+0xf2c/0x1000 Apr 17 12:27:40 zop03-01 kernel: [35640.185341] [c0000003a64b3c00] [c0000000002d45bc] path_openat+0xdc/0x810 Apr 17 12:27:40 zop03-01 kernel: [35640.185343] [c0000003a64b3cd0] [c0000000002d5c48] do_filp_open+0x58/0xf0 Apr 17 12:27:40 zop03-01 kernel: [35640.185346] [c0000003a64b3db0] [c0000000002bcbf8] do_sys_open+0x1c8/0x390 Apr 17 12:27:40 zop03-01 kernel: [35640.185348] [c0000003a64b3e30] [c000000000009258] syscall_exit+0x0/0x98 Apr 17 12:27:40 zop03-01 kernel: [35640.185350] INFO: task latest_blast:18549 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.185355] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.185358] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.185363] latest_blast D 00003fff9e03388c 0 18549 18377 0x00040000 Apr 17 12:27:40 zop03-01 kernel: [35640.185365] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.185367] [c0000003a64b7200] [c0000003cb0dc3e0] 0xc0000003cb0dc3e0 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.185369] [c0000003a64b73d0] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.185372] [c0000003a64b7420] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185374] [c0000003a64b7650] [c000000000a349a4] io_schedule+0xb4/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.185376] [c0000003a64b7680] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185378] [c0000003a64b76a0] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.185380] [c0000003a64b76f0] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.185382] [c0000003a64b7770] [c0000000003d3d30] do_get_write_access+0x3b0/0x670 Apr 17 12:27:40 zop03-01 kernel: [35640.185384] [c0000003a64b7850] [c0000000003d403c] jbd2_journal_get_write_access+0x4c/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185387] [c0000003a64b7880] [c0000000003b347c] __ext4_journal_get_write_access+0x8c/0xe0 Apr 17 12:27:40 zop03-01 kernel: [35640.185388] [c0000003a64b78c0] [c000000000372ba8] __ext4_new_inode+0x5d8/0x16d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185391] [c0000003a64b7a10] [c000000000387524] ext4_create+0x154/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185392] [c0000003a64b7ae0] [c0000000002d0798] vfs_create+0x178/0x210 Apr 17 12:27:40 zop03-01 kernel: [35640.185394] [c0000003a64b7b30] [c0000000002d175c] do_last+0xf2c/0x1000 Apr 17 12:27:40 zop03-01 kernel: [35640.185396] [c0000003a64b7c00] [c0000000002d45bc] path_openat+0xdc/0x810 Apr 17 12:27:40 zop03-01 kernel: [35640.185398] [c0000003a64b7cd0] [c0000000002d5c48] do_filp_open+0x58/0xf0 Apr 17 12:27:40 zop03-01 kernel: [35640.185401] [c0000003a64b7db0] [c0000000002bcbf8] do_sys_open+0x1c8/0x390 Apr 17 12:27:40 zop03-01 kernel: [35640.185403] [c0000003a64b7e30] [c000000000009258] syscall_exit+0x0/0x98 Apr 17 12:27:40 zop03-01 kernel: [35640.185405] INFO: task latest_blast:18550 blocked for more than 120 seconds. Apr 17 12:27:40 zop03-01 kernel: [35640.185410] Not tainted 3.19.0-9-generic #9-Ubuntu Apr 17 12:27:40 zop03-01 kernel: [35640.185413] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 17 12:27:40 zop03-01 kernel: [35640.185418] latest_blast D 00003fff9e032ce4 0 18550 18377 0x00040000 Apr 17 12:27:40 zop03-01 kernel: [35640.185420] Call Trace: Apr 17 12:27:40 zop03-01 kernel: [35640.185422] [c0000003a64af2a0] [c0000003a64af2e0] 0xc0000003a64af2e0 (unreliable) Apr 17 12:27:40 zop03-01 kernel: [35640.185425] [c0000003a64af470] [c000000000015a4c] __switch_to+0x1fc/0x350 Apr 17 12:27:40 zop03-01 kernel: [35640.185427] [c0000003a64af4c0] [c000000000a33f9c] __schedule+0x37c/0x8d0 Apr 17 12:27:40 zop03-01 kernel: [35640.185429] [c0000003a64af6f0] [c000000000a349a4] io_schedule+0xb4/0x130 Apr 17 12:27:40 zop03-01 kernel: [35640.185431] [c0000003a64af720] [c000000000a35454] bit_wait_io+0x44/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185433] [c0000003a64af740] [c000000000a34e84] __wait_on_bit+0xf4/0x170 Apr 17 12:27:40 zop03-01 kernel: [35640.185436] [c0000003a64af790] [c000000000a34f88] out_of_line_wait_on_bit+0x88/0xa0 Apr 17 12:27:40 zop03-01 kernel: [35640.185438] [c0000003a64af810] [c0000000003d3d30] do_get_write_access+0x3b0/0x670 Apr 17 12:27:40 zop03-01 kernel: [35640.185440] [c0000003a64af8f0] [c0000000003d403c] jbd2_journal_get_write_access+0x4c/0x80 Apr 17 12:27:40 zop03-01 kernel: [35640.185442] [c0000003a64af920] [c0000000003b347c] __ext4_journal_get_write_access+0x8c/0xe0 Apr 17 12:27:40 zop03-01 kernel: [35640.185444] [c0000003a64af960] [c00000000037af5c] ext4_reserve_inode_write+0xcc/0x100 Apr 17 12:27:40 zop03-01 kernel: [35640.185446] [c0000003a64af9b0] [c00000000037aff0] ext4_mark_inode_dirty+0x60/0x260 Apr 17 12:27:40 zop03-01 kernel: [35640.185448] [c0000003a64afa60] [c00000000037ecec] ext4_dirty_inode+0x5c/0x90 Apr 17 12:27:40 zop03-01 kernel: [35640.185450] [c0000003a64afa90] [c0000000002fdabc] __mark_inode_dirty+0x6c/0x340 Apr 17 12:27:40 zop03-01 kernel: [35640.185452] [c0000003a64afb10] [c0000000002e54ec] update_time+0xdc/0x150 Apr 17 12:27:40 zop03-01 kernel: [35640.185454] [c0000003a64afb50] [c0000000002e5828] file_update_time+0xd8/0x140 Apr 17 12:27:40 zop03-01 kernel: [35640.185457] [c0000003a64afba0] [c000000000214758] __generic_file_write_iter+0x278/0x430 Apr 17 12:27:40 zop03-01 kernel: [35640.185459] [c0000003a64afc00] [c00000000036ffe8] ext4_file_write_iter+0x118/0x480 Apr 17 12:27:40 zop03-01 kernel: [35640.185461] [c0000003a64afcc0] [c0000000002bd93c] new_sync_write+0xac/0x120 Apr 17 12:27:40 zop03-01 kernel: [35640.185462] [c0000003a64afd90] [c0000000002be4fc] vfs_write+0xdc/0x260 Apr 17 12:27:40 zop03-01 kernel: [35640.185465] [c0000003a64afde0] [c0000000002bf37c] SyS_write+0x6c/0x110 Apr 17 12:27:40 zop03-01 kernel: [35640.185467] [c0000003a64afe30] [c000000000009258] syscall_exit+0x0/0x98 Apr 17 12:28:11 zop03-01 systemd-udevd[18698]: timeout '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 5' Apr 17 12:28:11 zop03-01 systemd-udevd[18698]: slow: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 5' [18707] Apr 17 12:28:11 zop03-01 systemd-udevd[13255]: worker [18698] /devices/virtual/block/dm-5 timeout; kill it Apr 17 12:28:11 zop03-01 systemd-udevd[13255]: seq 3811 '/devices/virtual/block/dm-5' killed Apr 17 12:28:11 zop03-01 systemd-udevd[13255]: worker [18698] terminated by signal 9 (Killed) Apr 17 12:28:13 zop03-01 systemd-udevd[18712]: timeout '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 24' Apr 17 12:28:13 zop03-01 systemd-udevd[18712]: slow: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 24' [18738] Apr 17 12:28:13 zop03-01 systemd-udevd[18711]: timeout '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 23' Apr 17 12:28:13 zop03-01 systemd-udevd[18711]: slow: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 23' [18737] Apr 17 12:28:14 zop03-01 systemd-udevd[18712]: timeout: killing '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 24' [18738] Apr 17 12:28:14 zop03-01 systemd-udevd[18712]: slow: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 24' [18738] Apr 17 12:28:14 zop03-01 systemd-udevd[18712]: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 24' [18738] terminated by signal 9 (Killed) Apr 17 12:28:14 zop03-01 systemd-udevd[18712]: timeout '/sbin/multipath -v0 /dev/dm-24' Apr 17 12:28:14 zop03-01 systemd-udevd[18712]: slow: '/sbin/multipath -v0 /dev/dm-24' [19553] Apr 17 12:28:14 zop03-01 systemd-udevd[18712]: conflicting device node '/dev/mapper/mpath5' found, link to '/dev/dm-24' will not be created Apr 17 12:28:14 zop03-01 systemd-udevd[18711]: timeout: killing '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 23' [18737] Apr 17 12:28:14 zop03-01 systemd-udevd[18711]: slow: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 23' [18737] Apr 17 12:28:14 zop03-01 systemd-udevd[18711]: '/sbin/dmsetup ls --target multipath --exec '/sbin/kpartx -a -p -part' -j 252 -m 23' [18737] terminated by signal 9 (Killed) Apr 17 12:28:14 zop03-01 systemd-udevd[18711]: timeout '/sbin/multipath -v0 /dev/dm-23' Apr 17 12:28:14 zop03-01 systemd-udevd[18711]: slow: '/sbin/multipath -v0 /dev/dm-23' [19564] Apr 17 12:28:14 zop03-01 systemd-udevd[18711]: conflicting device node '/dev/mapper/mpath9' found, link to '/dev/dm-23' will not be created Apr 17 12:28:16 zop03-01 kernel: [35676.081347] sd 2:0:1:0: [sdf] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:16 zop03-01 kernel: [35676.081359] sd 2:0:1:0: [sdf] Sense Key : Unit Attention [current] Apr 17 12:28:16 zop03-01 kernel: [35676.081365] sd 2:0:1:0: [sdf] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:16 zop03-01 kernel: [35676.081369] sd 2:0:1:0: [sdf] CDB: Apr 17 12:28:16 zop03-01 kernel: [35676.081371] Read(10): 28 00 00 00 00 00 00 00 08 00 Apr 17 12:28:16 zop03-01 kernel: [35676.081382] blk_update_request: I/O error, dev sdf, sector 0 Apr 17 12:28:16 zop03-01 kernel: [35676.264902] sd 2:0:1:4: [sdj] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:16 zop03-01 kernel: [35676.264910] sd 2:0:1:4: [sdj] Sense Key : Unit Attention [current] Apr 17 12:28:16 zop03-01 kernel: [35676.264914] sd 2:0:1:4: [sdj] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:16 zop03-01 kernel: [35676.264917] sd 2:0:1:4: [sdj] CDB: Apr 17 12:28:16 zop03-01 kernel: [35676.264921] Read(10): 28 00 00 00 00 00 00 00 08 00 Apr 17 12:28:16 zop03-01 kernel: [35676.264929] blk_update_request: I/O error, dev sdj, sector 0 Apr 17 12:28:16 zop03-01 kernel: [35676.264938] sd 2:0:1:1: [sdg] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:16 zop03-01 kernel: [35676.264940] sd 2:0:1:1: [sdg] Sense Key : Unit Attention [current] Apr 17 12:28:16 zop03-01 kernel: [35676.264942] sd 2:0:1:1: [sdg] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:16 zop03-01 kernel: [35676.264942] sd 2:0:1:1: [sdg] CDB: Apr 17 12:28:16 zop03-01 kernel: [35676.264946] Read(10): 28 00 00 00 00 00 00 00 08 00 Apr 17 12:28:16 zop03-01 kernel: [35676.264951] blk_update_request: I/O error, dev sdg, sector 0 Apr 17 12:28:16 zop03-01 kernel: [35676.264966] sd 2:0:1:2: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:16 zop03-01 kernel: [35676.264970] sd 2:0:1:2: [sdh] Sense Key : Unit Attention [current] Apr 17 12:28:16 zop03-01 kernel: [35676.264977] sd 2:0:1:2: [sdh] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:16 zop03-01 kernel: [35676.264981] sd 2:0:1:2: [sdh] CDB: Apr 17 12:28:16 zop03-01 kernel: [35676.264983] Read(10): 28 00 00 00 00 00 00 00 08 00 Apr 17 12:28:16 zop03-01 kernel: [35676.264992] blk_update_request: I/O error, dev sdh, sector 0 Apr 17 12:28:16 zop03-01 kernel: [35676.265005] sd 2:0:1:3: [sdi] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:16 zop03-01 kernel: [35676.265009] sd 2:0:1:3: [sdi] Sense Key : Unit Attention [current] Apr 17 12:28:16 zop03-01 kernel: [35676.265013] sd 2:0:1:3: [sdi] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:16 zop03-01 kernel: [35676.265016] sd 2:0:1:3: [sdi] CDB: Apr 17 12:28:16 zop03-01 kernel: [35676.265018] Read(10): 28 00 00 00 00 00 00 00 08 00 Apr 17 12:28:16 zop03-01 kernel: [35676.265026] blk_update_request: I/O error, dev sdi, sector 0 Apr 17 12:28:17 zop03-01 multipathd: mpath3: sdf - directio checker reports path is up Apr 17 12:28:17 zop03-01 multipathd: 8:80: reinstated Apr 17 12:28:17 zop03-01 multipathd: mpath3: remaining active paths: 3 Apr 17 12:28:17 zop03-01 multipathd: mpath0: sdg - directio checker reports path is up Apr 17 12:28:17 zop03-01 multipathd: 8:96: reinstated Apr 17 12:28:17 zop03-01 multipathd: mpath0: remaining active paths: 3 Apr 17 12:28:17 zop03-01 multipathd: mpath4: sdh - directio checker reports path is down Apr 17 12:28:17 zop03-01 multipathd: mpath1: sdi - directio checker reports path is down Apr 17 12:28:17 zop03-01 multipathd: mpath2: sdj - directio checker reports path is down Apr 17 12:28:17 zop03-01 multipathd: mpath7: sdw - directio checker reports path is down Apr 17 12:28:17 zop03-01 multipathd: mpath8: sdx - directio checker reports path is down Apr 17 12:28:17 zop03-01 multipathd: mpath9: sdy - directio checker reports path is down Apr 17 12:28:17 zop03-01 multipathd: mpath5: sdz - directio checker reports path is down Apr 17 12:28:17 zop03-01 multipathd: mpath6: sdaa - directio checker reports path is down Apr 17 12:28:17 zop03-01 systemd-udevd[18747]: conflicting device node '/dev/mapper/mpath0' found, link to '/dev/dm-21' will not be created Apr 17 12:28:17 zop03-01 systemd-udevd[18788]: conflicting device node '/dev/mapper/mpath3-part2' found, link to '/dev/dm-15' will not be created Apr 17 12:28:17 zop03-01 systemd-udevd[18786]: conflicting device node '/dev/mapper/mpath3-part1' found, link to '/dev/dm-14' will not be created Apr 17 12:28:17 zop03-01 systemd-udevd[18789]: conflicting device node '/dev/mapper/mpath3-part3' found, link to '/dev/dm-16' will not be created Apr 17 12:28:17 zop03-01 systemd-udevd[18746]: conflicting device node '/dev/mapper/mpath0-part1' found, link to '/dev/dm-26' will not be created Apr 17 12:25:14 zop03-01 systemd-udevd[18796]: conflicting device node '/dev/mapper/mpath4-part1' found, link to '/dev/dm-25' will not be created Apr 17 12:28:17 zop03-01 systemd-udevd[18796]: conflicting device node '/dev/mapper/mpath0-part2' found, link to '/dev/dm-28' will not be created Apr 17 12:28:17 zop03-01 systemd-udevd[18854]: conflicting device node '/dev/mapper/mpath0-part3' found, link to '/dev/dm-31' will not be created Apr 17 12:28:17 zop03-01 kernel: [35677.253493] sd 4:0:0:4: [sdaa] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:17 zop03-01 kernel: [35677.253503] sd 4:0:0:5: [sdab] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:17 zop03-01 kernel: [35677.253507] sd 4:0:0:4: [sdaa] Sense Key : Unit Attention [current] Apr 17 12:28:17 zop03-01 kernel: [35677.253511] sd 4:0:0:5: [sdab] Sense Key : Unit Attention [current] Apr 17 12:28:17 zop03-01 kernel: [35677.253514] sd 4:0:0:4: [sdaa] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:17 zop03-01 kernel: [35677.253515] sd 4:0:0:5: [sdab] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:17 zop03-01 kernel: [35677.253518] sd 4:0:0:5: [sdab] CDB: Apr 17 12:28:17 zop03-01 kernel: [35677.253520] sd 4:0:0:4: [sdaa] CDB: Apr 17 12:28:17 zop03-01 kernel: [35677.253521] Write(10): 2a 00 02 Apr 17 12:28:17 zop03-01 kernel: [35677.253527] Read(10) c7 15 Apr 17 12:28:17 zop03-01 kernel: [35677.253530] : 28 Apr 17 12:28:17 zop03-01 kernel: [35677.253532] 38 00 04 00 Apr 17 12:28:17 zop03-01 kernel: [35677.253536] 00 00 Apr 17 12:28:17 zop03-01 kernel: [35677.253540] 00 Apr 17 12:28:17 zop03-01 kernel: [35677.253541] 00 Apr 17 12:28:17 zop03-01 kernel: [35677.253544] blk_update_request: I/O error, dev sdab, sector 46601528 Apr 17 12:28:17 zop03-01 kernel: [35677.253547] 00 00 00 00 08 00 Apr 17 12:28:17 zop03-01 kernel: [35677.253550] blk_update_request: I/O error, dev sdaa, sector 0 Apr 17 12:28:17 zop03-01 kernel: [35677.253579] sd 4:0:0:2: [sdy] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:17 zop03-01 kernel: [35677.253584] sd 4:0:0:2: [sdy] Sense Key : Unit Attention [current] Apr 17 12:28:17 zop03-01 kernel: [35677.253588] sd 4:0:0:2: [sdy] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:17 zop03-01 kernel: [35677.253591] sd 4:0:0:2: [sdy] CDB: Apr 17 12:28:17 zop03-01 kernel: [35677.253593] Read(10): 28 00 00 00 00 00 00 00 08 00 Apr 17 12:28:17 zop03-01 kernel: [35677.253609] blk_update_request: I/O error, dev sdy, sector 0 Apr 17 12:28:17 zop03-01 kernel: [35677.253636] sd 4:0:0:3: [sdz] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:17 zop03-01 kernel: [35677.253640] sd 4:0:0:3: [sdz] Sense Key : Unit Attention [current] Apr 17 12:28:17 zop03-01 kernel: [35677.253643] sd 4:0:0:3: [sdz] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:17 zop03-01 kernel: [35677.253647] sd 4:0:0:3: [sdz] CDB: Apr 17 12:28:17 zop03-01 kernel: [35677.253649] Read(10): 28 00 00 00 00 00 00 00 08 00 Apr 17 12:28:17 zop03-01 kernel: [35677.253658] blk_update_request: I/O error, dev sdz, sector 0 Apr 17 12:28:17 zop03-01 kernel: [35677.253672] sd 4:0:0:0: [sdw] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Apr 17 12:28:17 zop03-01 kernel: [35677.253675] sd 4:0:0:0: [sdw] Sense Key : Unit Attention [current] Apr 17 12:28:17 zop03-01 kernel: [35677.253678] sd 4:0:0:0: [sdw] Add. Sense: Power on, reset, or bus device reset occurred Apr 17 12:28:17 zop03-01 kernel: [35677.253681] sd 4:0:0:0: [sdw] CDB: Apr 17 12:28:17 zop03-01 kernel: [35677.253683] Read(10): 28 00 00 00 00 00 00 00 08 00 Apr 17 12:28:17 zop03-01 kernel: [35677.253690] blk_update_request: I/O error, dev sdw, sector 0 root@zop03-01:~#