iSCSI SAN

[Julien Cigar]

On Mon, Apr 26, 2021 at 05:19:14PM +0200, joris dedieu wrote:
> Hi,
> 
> Le lun. 26 avr. 2021 à 14:24, Julien Cigar <julien at perdition.city> a écrit :
> 
> > Hello,
> >
> > I'm wondering if something has already been written to implement a
> > fully redundant and highly available FreeBSD ZFS based iSCSI SAN?
> >
> 
> > I've setup some FreeBSD iSCSI SAN-like in the past (for small
> > structures) and it has always worked well. However upgrades have always
> > been painfull and, although there is ZFS, redundant power supplies,
> > redundant switches with multipath, it's still a SPOF if a non-redundant
> > component dies, like the motherboard. It's not like an HPE MSA-like
> > system where everything is redundant out of the box.
> >
> 
> From my experience HA mechanisms are an infinite source of pain. Don't

>From my experence "automatic failover" is an infinite source of pain,
not HA

> forget the power of simplicity. What uptime do you get with your simple
> setup ? When was the last time you see a decent mainboard crash ?

As long as I don't touch at it everything is fine. Now, as soon as I 
have to reboot the server or upgrade (from example from one major 
release to another) there are long minutes of downtime, which is
unacceptable if many servers are connected.

> Also don't forget that FreeBSD has glusterfs and ceph.
> 

I haven't looked at CEPH yet, but I don't think everything has been
ported (KRDB, CEPH native fs, etc). It is certainly not "production
ready".

> If you want to do something similar to proprietary chassis, you should have
> to look at SAS HBA and JBOD chassis, ATAoE chassis (if it still exists) or

Yes, definitively. SAS HBA with a JBOD chassis is something I was
looking too .. unfortunately I don't have the hardware to experiment :(

> stuff like that to attach your disks to your two mainboard. Still (OMG)
> dealing with zfs (import -f) on failover, cluster STONITH and other voodoo.
> You will have fully redundant design (see
> https://i.stack.imgur.com/ijjpk.png )
> 
> Cheers
> Joris
> 

Thanks,
Julien

> 
> > So the idea came to me for the iSCSI target to setup 2 physical servers
> > with a bunch of disks, create some raidzx on them and export one ZFS
> > volume per initiator on each target, a bit like on (1)
> >
> 
> > I've tried to setup that in a small "lab", with some jails, gmultipath,
> > two switches, and several VLANs. Unfortunately no 10 gbits to test, but
> > 3x1Gbits LAGG with LACP.
> >
> > The downside of this setup is that "half" of the storage is (temporarily)
> > lost when a target reboots (freebsd-update, upgrades, etc), which
> > de-facto disqualifies gmirror + UFS on the initiator side as a full
> > resync of required and takes ages. With ZFS you don't have this problem
> > as only the delta is resync. For the few tests I did it seems to work,
> > but I'm wondering: zfs over zvol .. is it sane? does it makes sense?
> > could I disable checksum on the initiator side to speed up things? do
> > you see any race condition or ... with this setup (sync, etc)?
> >
> > What do you think?
> >
> > Thanks!
> > Julien
> >
> > (1) https://gist.github.com/silenius/c6d1020aca54c47f71aa9f2a19a55ffe
> >
> >
> > --
> > Julien Cigar
> > Belgian Biodiversity Platform (http://www.biodiversity.be)
> > PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
> > No trees were killed in the creation of this message.
> > However, many electrons were terribly inconvenienced.
> > _______________________________________________
> > freebsd-fs at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> > To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"
> >
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"

-- 
Julien Cigar
Belgian Biodiversity Platform (http://www.biodiversity.be)
PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.