ZFS on Hardware RAID

[Steven Hartland]

On 22/01/2019 11:52, Borja Marcos wrote:
>
>> On 21 Jan 2019, at 16:40, Steven Hartland <killing at multiplay.co.uk> wrote:
>>
>> Typically yes it does.
>>> The same could be said of other functionalities available when you talk directly
>>> to the disks.
>>>
>>>
>>>
>>>> That's why I did as I describe above.  I didn't have the money to buy an
>>>> HBA until a year or two later, and I had good backups, so I used the PERC
>>>> 6/e.
>>>>
>>> Depending on the LSI model used in that card you can make it behave like an HBA.
>>>
>>>
>> Indeed you can for most, but its not an easy or error free option, requires force flashing from IR to IT firmware, and if it goes wrong you can end up with a bricked controller.
>>
>> We've used LSI's in IR mode (RAID) but as individual disks JBOD with ZFS raid2z or mirroring on top for a long time and on lots of machines, not had any issues to report. So while I would never use hardware RAID from those controllers with ZFS, JBOD on them is an easy and reliable option in our experience.
> Sorry to insist, but maybe we are talking of different stuff? My apologies in advance if I am confused and/or my
> information is outdated.
>
> JBOD mode: a mode offered by some LSI Logic IR controllers to create virtual devices mapped so that each one is effectively.
> For example, with the mfi driver the “disks” appear as “mfisyspd” devices while the “conventional” logical volumes appear as “mfid”. My information could be outdated though?
Cards that use mfi are quite a different beasts, they are more a pure 
RAID controller, than cards which have both IT & IR modes, where the 
RAID capabilities are more limited.

The later present as daX devices where the former presents as mfid 
unless you force otherwise using mfi/mfip as you detail below.

> HBA mode: when the card is in IT mode *or* it does expose the actual targets to the CAM layer. In the past I did it
> by manually patching the drivers and I’ve kept systems running smoothly for many years despite using IR cards.
> Currently you can use hw.mfi.allow_cam_disk_passthrough tunable to achieve the same effect without ugly tinkering.
>
> Unless I am terribly wrong, when I tried the JBOD mode as defined above (with LSI SAS3 cards and IR firmware) the mfisyspd
> devices were not actual CAM devices.
>
> Actually I find “JBOD” an artificial and confusing term because it can mean so many things. I would have rather preferred
> LSI to avoid it and provide either real, transparent access to the actual targets insteaf of somewhat “virtualizing” them.
>
> Again, I may be confused? I haven’t tried to configure a LSI IR card as “JBOD” for a long time. Maybe now they do
> expose the targets instead of defining logical volumes?
I can definitely confirm LSI controllers which present individual daX 
devices can support TRIM / UMAP based off what the device reports.

As you say its quite confusing, so hope that clarifies?

     Regards
     Steve