ZFS on Hardware RAID

[Borja Marcos]

> On 22 Jan 2019, at 16:21, Steven Hartland <killing at multiplay.co.uk> wrote:
> 
> 
>> Sorry to insist, but maybe we are talking of different stuff? My apologies in advance if I am confused and/or my
>> information is outdated.
>> 
>> JBOD mode: a mode offered by some LSI Logic IR controllers to create virtual devices mapped so that each one is effectively.
>> For example, with the mfi driver the “disks” appear as “mfisyspd” devices while the “conventional” logical volumes appear as “mfid”. My information could be outdated though?
> Cards that use mfi are quite a different beasts, they are more a pure RAID controller, than cards which have both IT & IR modes, where the RAID capabilities are more limited.

Yes, that’s right. 

Going to actual examples, which I hope will be useful:

- mps cards (LSI2008) can run either IT or IR firmware. Both present disks as pure targets. 

- MFI and others, however, which are pure RAID cards, are the pathological case I mentioned (the so called JBOD mode is deceiving). 
For those you can force the driver to really present da (CAM) targets bypassing the RAID logic. I have done it several times and one
of my servers has been doing this since 2007.

- To make it even more confusing, you can choose the driver for LSI SAS3 cards. You can use MFI or the new MRSAS driver. The latter
should offer the physical disks as regular CAM targets.

> The later present as daX devices where the former presents as mfid unless you force otherwise using mfi/mfip as you detail below.

Yes.

Speaking of IT and IR, Stephen Mcconnell from LSI/Avago said that the performance difference should be really negligible. Of course it’s safer
to run IT firmware avoiding the possibility of making a mistake with the configuration. 

>> HBA mode: when the card is in IT mode *or* it does expose the actual targets to the CAM layer. In the past I did it
>> by manually patching the drivers and I’ve kept systems running smoothly for many years despite using IR cards.
>> Currently you can use hw.mfi.allow_cam_disk_passthrough tunable to achieve the same effect without ugly tinkering.
>> 
>> Unless I am terribly wrong, when I tried the JBOD mode as defined above (with LSI SAS3 cards and IR firmware) the mfisyspd
>> devices were not actual CAM devices.
>> 
>> Actually I find “JBOD” an artificial and confusing term because it can mean so many things. I would have rather preferred
>> LSI to avoid it and provide either real, transparent access to the actual targets insteaf of somewhat “virtualizing” them.
>> 
>> Again, I may be confused? I haven’t tried to configure a LSI IR card as “JBOD” for a long time. Maybe now they do
>> expose the targets instead of defining logical volumes?
> I can definitely confirm LSI controllers which present individual daX devices can support TRIM / UMAP based off what the device reports.

Yes, mps and mpr.

MFI includes a not widely publicized tunable (allow_cam_passthrough) and for aac (although these are really old) it’s possible to hack the
driver (aac_cam.c) so that disks are reported to the CAM layer.


> As you say its quite confusing, so hope that clarifies?

I think so, I was trying to avoid confusion for newcomers, not intending to be fastidious. The JBOD term is ambiguous. This stuff drove me mad some time ago, not
to mention the puzzled faces of the Dell salesmen when I said “EGGS, BACON, RAID, RAID, RAID, I DON’T LIKE RAID!” ;)

They thought I was nuts rejecting a “hardware” RAID. And in some cases they almost refuse to sell servers with plain HBAs. 





Borja.