Odd gmultipath behaviour

[Trent Nelson]

This seems... odd:

[root at s16/ttypts/1(../misc/isp/bin)#] gmultipath status                         
            Name   Status  Components
multipath/292_61   BROKEN  N/A
multipath/292_39   BROKEN  N/A
multipath/292_43   BROKEN  N/A
multipath/292_54   BROKEN  N/A
multipath/292_39  OPTIMAL  da0 (ACTIVE)
                           da3 (PASSIVE)
                           da8 (PASSIVE)
                           da11 (PASSIVE)
multipath/292_43  OPTIMAL  da1 (ACTIVE)
                           da5 (PASSIVE)
                           da12 (PASSIVE)
                           da15 (PASSIVE)
multipath/292_61  OPTIMAL  da2 (ACTIVE)
                           da6 (PASSIVE)
                           da9 (PASSIVE)
                           da13 (PASSIVE)
multipath/292_54  OPTIMAL  da4 (ACTIVE)
                           da7 (PASSIVE)
                           da10 (PASSIVE)
                           da14 (PASSIVE)

I was playing around with isp's target mode, zfs and ctladm.  I've got two dual-port HBAs, I set one of each port to initiator and the other to target and created a zfs pool (mirror 292_61 292_39 mirror 292_43 292_54).

Everything was fine up to this point.  Then I ran a few ctladm commands, the last one being `ctladm port -l`, then things started going a bit screwy.  I ran Matt's isp/tools/isp_getrole and all four HBAs were now purportedly in target mode.

That, understandably, hosed the gmultipath setup, which resulted in my zpool hanging.  I manually isp/tools/isp_setrole'd all the HBAs back to initiator, isp/tools/isp_reset them, then `camcontrol rescan all`'d, which brought everything back.

But, uh, what's going on with gmultipath?  Why's it hanging on to the broken paths?

Some more output:

[root at s16/ttypts/1(../misc/isp/bin)#] gmultipath list   
Geom name: 292_61
Type: AUTOMATIC
Mode: Active/Passive
UUID: 1ad76fac-cf53-11e1-903e-00118513ae7d
State: BROKEN
Providers:
1. Name: multipath/292_61
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: BROKEN

Geom name: 292_39
Type: AUTOMATIC
Mode: Active/Passive
UUID: 65b3e0aa-cf53-11e1-903e-00118513ae7d
State: BROKEN
Providers:
1. Name: multipath/292_39
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: BROKEN

Geom name: 292_43
Type: AUTOMATIC
Mode: Active/Passive
UUID: b5f58ec4-cf53-11e1-903e-00118513ae7d
State: BROKEN
Providers:
1. Name: multipath/292_43
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: BROKEN

Geom name: 292_54
Type: AUTOMATIC
Mode: Active/Passive
UUID: ef54b03c-cf53-11e1-903e-00118513ae7d
State: BROKEN
Providers:
1. Name: multipath/292_54
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: BROKEN

Geom name: 292_39
Type: AUTOMATIC
Mode: Active/Passive
UUID: 65b3e0aa-cf53-11e1-903e-00118513ae7d
State: OPTIMAL
Providers:
1. Name: multipath/292_39
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r0w0e0
   State: OPTIMAL
Consumers:
1. Name: da0
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: ACTIVE
2. Name: da3
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE
3. Name: da8
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE
4. Name: da11
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE

Geom name: 292_43
Type: AUTOMATIC
Mode: Active/Passive
UUID: b5f58ec4-cf53-11e1-903e-00118513ae7d
State: OPTIMAL
Providers:
1. Name: multipath/292_43
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r0w0e0
   State: OPTIMAL
Consumers:
1. Name: da1
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: ACTIVE
2. Name: da5
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE
3. Name: da12
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE
4. Name: da15
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE

Geom name: 292_61
Type: AUTOMATIC
Mode: Active/Passive
UUID: 1ad76fac-cf53-11e1-903e-00118513ae7d
State: OPTIMAL
Providers:
1. Name: multipath/292_61
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r0w0e0
   State: OPTIMAL
Consumers:
1. Name: da2
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: ACTIVE
2. Name: da6
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE
3. Name: da9
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE
4. Name: da13
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE

Geom name: 292_54
Type: AUTOMATIC
Mode: Active/Passive
UUID: ef54b03c-cf53-11e1-903e-00118513ae7d
State: OPTIMAL
Providers:
1. Name: multipath/292_54
   Mediasize: 146815737344 (136G)
   Sectorsize: 512
   Mode: r0w0e0
   State: OPTIMAL
Consumers:
1. Name: da4
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: ACTIVE
2. Name: da7
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE
3. Name: da10
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE
4. Name: da14
   Mediasize: 146815737856 (136G)
   Sectorsize: 512
   Mode: r1w1e1
   State: PASSIVE


Thoughts?

	Trent.