HAST + ZFS + NFS + CARP

[Jan Bramkamp]

On 04/07/16 19:55, Jordan Hubbard wrote:
>
>> On Jul 3, 2016, at 11:05 PM, Ben RUBSON <ben.rubson at gmail.com> wrote:
>>
>> Of course Jordan, in this topic, we (well at least me :) make the following assumption :
>> one iSCSI target/disk = one real physical disk (a SAS disk, a SSD disk...), from a server having its own JBOD, no RAID adapter or whatever, just what ZFS likes !
>
> I certainly wouldn’t make that assumption.  Once you allow iSCSI to be the back-end in any solution, end-users will avail themselves of the flexibility to also export arbitrary or synthetic devices (like zvols / RAID devices) as “disks”.  You can’t stop them from doing so, so you might as well incorporate that scenario into your design.  Even if you could somehow enforce the 1:1 mapping of LUN to disk, iSCSI itself is still going to impose a serialization / performance / reporting (iSCSI LUNs don’t report SMART status) penalty that removes a lot of the advantages of having direct physical access to the media, so one might also ask what you’re gaining by imposing those restrictions.


How about 3way ZFS mirrors spread over three SAS JBODs with dual-ported 
expanders connected to two FreeBSD servers with SAS HBAs and a 
*reliable* arbiter to the disks. This could either be an external 
locking server e.g. consul/etcd/zookeeper and/or SCSI reservations. If 
more than two head servers are to share the disks a pair of SAS switches 
should do the job.

If N-1 disk redundancy is enough two JBODs and 2way mirrors would work 
as well.

While you can't prevent stupid operators from blowing their feet of it 
doesn't offer the same "flexibility" as iSCSI if only because you can't 
conveniently hookup everything talking Ethernet offering itself als 
iSCSI target. That is until someone implements a SAS target with CTL and 
a suitable HBA in FreeBSD ;-).

This kind of setup should also preserve all assumptions ZFS has 
regarding disks.

I have the required spare hardware to build a two JBOD test setup [1] 
and could run some tests if anyone is interested in such a setup.


[1]: Test setup

     +-----------+    +-----------+
     | MASTER    |    | SLAVE     |
     |           |    |           |
     | HBA0 HBA1 |    | HBA0 HBA1 |
     +--+----+---+    +--+----+---+
        ^    ^           ^    ^
        |    |           |    |
        |    |           |    +------+
        |    |           |           |
        |    |           +----+      |
        |    |                |      |
        |    +-----------+    |      |
        |                |    |      |
        v                v    v      |
     +--+--------+    +--+----+---+  |
     | JBOD 0    |    | JBOD 1    |  |
     +-------+---+    +-----------+  |
             ^                       |
             |                       |
             +-----------------------+