kern/178473: gmultipath needs to be smarter about errors

[Garrett Wollman]

>Number:         178473
>Category:       kern
>Synopsis:       gmultipath needs to be smarter about errors
>Confidential:   no
>Severity:       non-critical
>Priority:       medium
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          change-request
>Submitter-Id:   current-users
>Arrival-Date:   Fri May 10 07:20:00 UTC 2013
>Closed-Date:
>Last-Modified:
>Originator:     Garrett Wollman
>Release:        FreeBSD 9.1-RELEASE-p2 amd64
>Organization:
MIT Computer Science & Artificial Intelligence Laboratory
>Environment:
System: FreeBSD xyz.csail.mit.edu 9.1-RELEASE-p2 FreeBSD 9.1-RELEASE-p2 #27 r250065M: Mon Apr 29 17:11:15 EDT 2013 wollman at xyz.csail.mit.edu:/usr/obj/usr/src/sys/CSAIL amd64

>Description:

Currently, hard medium errors reported by the underlying provider will
cause geom_multipath to cycle infinitely, turning what should be a
failure reported to the consumer (ZFS in my case, which can do
something useful with it) into a write that simply never returns at
all, untill the hardware is physically offlined.

It looks like this:

(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x2103f4f7
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x2103f4f7
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040687
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040687
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040f4f
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040f4f
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 43 fe 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x210443ff
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 43 fe 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x210443ff
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 ac 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x2103f4f7
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 ac 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x2103f4f7
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040687
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040687
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040f4f
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040f4f
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 17 f3 0 0 2a 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21041817
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 67 eb 0 0 2a 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x210467ec
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 67 eb 0 0 2a 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x210467ec
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 d9 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x2103f4f7
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 d9 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x2103f4f7
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040687
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040687
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040f4f
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040f4f
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x2103f4f7
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x2103f4f7
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040687
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040687
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 e 7d 0 1 0 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040f4f
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 e 7d 0 1 0 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040f4f
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
pid 62004 (ntpd), uid 0: exited on signal 11 (core dumped)
pid 62021 (ntpd), uid 0: exited on signal 11 (core dumped)
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x2103f4f7
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 3 f4 58 0 0 d6 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x2103f4f7
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040687
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040687
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040f4f
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040f4f
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040687
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 6 5d 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040687
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12
(da158:mps2:0:77:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da158:mps2:0:77:0): CAM status: SCSI Status Error
(da158:mps2:0:77:0): SCSI status: Check Condition
(da158:mps2:0:77:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da158:mps2:0:77:0): Info: 0x21040f4f
(da158:mps2:0:77:0): Actual Retry Count: 63
(da158:mps2:0:77:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da158 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da85
GEOM_MULTIPATH: da85 is now active path in s25d12
(da85:mps1:0:102:0): READ(10). CDB: 28 0 21 4 f 28 0 0 2b 0 
(da85:mps1:0:102:0): CAM status: SCSI Status Error
(da85:mps1:0:102:0): SCSI status: Check Condition
(da85:mps1:0:102:0): SCSI sense: MEDIUM ERROR asc:11,1 (Read retries exhausted)
(da85:mps1:0:102:0): Info: 0x21040f4f
(da85:mps1:0:102:0): Actual Retry Count: 63
(da85:mps1:0:102:0): Error 5, Unretryable error
GEOM_MULTIPATH: Error 5, da85 in s25d12 marked FAIL
GEOM_MULTIPATH: all paths in s25d12 were marked FAIL, restore da158
GEOM_MULTIPATH: da158 is now active path in s25d12

>How-To-Repeat:

Build a big multipathed zpool, and use it.  Eventually, a disk will
fail.  Wonder why ZFS doesn't notice it.  (Apparently the SMART
implementation on the drive doesn't consider a single hard read error
to be "bad enough" to raise the failure-imminent bit.)  Finally get
the answer in the nightly root mail.

>Fix:

Don't know.  geom_multipath should have some way of figuring out that
a failure is "really" a failure on the underlying device, and notify
the consumer appropriately.  Perhaps keep some state in the bio that
remembers which providers have failed the request already, and if
every available provider fails the request without any intervening
configuration changes, then return the last failure to the consumer.
>Release-Note:
>Audit-Trail:
>Unformatted: