Best practice for high availability ZFS pool
Jan Bramkamp
crest at rlwinm.de
Tue May 17 09:56:41 UTC 2016
On 16/05/16 12:08, Palle Girgensohn wrote:
> Hi,
>
> We need to set up a ZFS pool with redundance. The main goal is high availability - uptime.
>
> I can see a few of paths to follow.
>
> 1. HAST + ZFS
>
> 2. Some sort of shared storage, two machines sharing a JBOD box.
If you're willing to put your disks into JBODs you can use JBODs with
two upstream ports per SAS expander and hook up one port to each head
node. Now you can access the all disks on both head nodes. The next step
you require is reliable master election. Two nodes alone can't form the
required consensus. In theory you could use SCSI persistent
reservations, but afaik FreeBSD lacks the tooling unless you want send
raw SCSI commands through camcontrol. The easier solution is to run some
master election using one or three additional nodes for a total of three
or five. Both consul and etcd are available as ports and are designed
for reliable master election without special hardware.
If you go done this way you still need some kind of fencing (maybe via
IPMI or PDUs).
Now the JBOD is your SPoF so get yourself at least two or better three
JBODs. For optimal performance and reliability use three JBODs with
3-way mirrors spread over all JOBDs.
In this setup no hardware protects your disks from the hot standby. If
it falls out of sync you have to keep it from writing to the shared
directed attached storage. One way to achieve this would be to load the
SAS HBA kernel module only after the role (primary, backup) has been
elected and disable the HBA option ROM in the UEFI/BIOS.
I tried this once out of curiosity and it performed well, but good look
finding any support for such a setup.
The same kind of setup should be possible with iSCSI instead of SAS
disks connected to dual ported expanders, but I can't say anything about
the performance you can expect from the FreeBSD iSCSI target and
initiator. At least it would simply fencing a lot because the fencing
could be moved from the SCSI initiators into the SCSI targets.
Jan Bramkamp
More information about the freebsd-fs
mailing list