Fun with HAST and inter-host connections

[Julien Cigar]

On Thu, Aug 31, 2017 at 01:33:30PM +0200, Julien Cigar wrote:
> On Wed, Aug 30, 2017 at 10:40:03PM +0100, Frank Leonhardt wrote:
> > Please note - I'm pushing what can be done with commodity hardware for 
> > amusement here, and trying to do interesting things with FreeBSD. I *do* 
> > want to do it this way.
> > 
> > Okay, so the game is this: I've got some identical hosts with disks and 
> > I've been working on ways of clustering them for a long time. It's what 
> > I do for fun, right?
> > 
> > I'm currently playing with failover storage. Not necessarily with HAST; 
> > iSCSI is fun and I'm messing with geom stuff in general. But let's stick 
> > with HAST as it illustrates the dilemma.
> > 
> > When I started this game, 1Gb Ethernet was blistering. Now it's not so 
> > hot. How do I "network" the hosts with as much throughput as possible 
> > (with IP sockets, preferably)? Options:
> > 
> > 1) 10Gb Ethernet is expensive. 10Gb switches even more so, but I can do 
> > point-to-point.
> > 
> > 2) LAG is more about failover than speed. And anyway, you end up needing 
> > a lot of Ethernet ports on each host and it soon gets crazy.
> > 
> > 3) Fibrechannel - if I bought a few old (cheap) fiberchannel cards, I'm 
> > not sure how I could use them point-to-point. Is this possible and has 
> > anyone done it?
> > 
> > 4) USB 3.1. 10Gb. PCIe cards cost about $30. Now this sounds fun. IP 
> > over USB anyone?
> > 
> > Now please don't advise me to get a pair of fabric switches and do the 
> > job properly. That's not what this is about. I want to see if it's 
> > possible to make a fast(er) storage solution using cheap components. A 
> > sort of Redundant Array of Inexpensive NAS.
> > 
> > Thoughts anyone? In particular, is the USB 3.1 idea crazy? And is anyone 
> > else crazy enough to be trying the same thing?
> 
> I've set up a low-cost redundant storage here at work with ZFS + iSCSI,
> basically a zpool mirror over 2 local disks and 2 iSCSI disks. Failover
> is done through CARP and devd. It works well (except when Jumbo Frames
> are turned on) for almost a year now. Just for extra security, and also
> because some people (smarter than me in this area) advised me against 
> this approach, I've setup a third node with ZFS replication (zrep).
> Basically if you can afford to lose "some" data, I would suggest to go
> with ZFS replication.

(of course by "ZFS replication" I mean snapshot and send/receive)

> 
> > 
> > Thanks, Frank.
> > 
> > 
> > _______________________________________________
> > freebsd-questions at freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> > To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"
> 
> -- 
> Julien Cigar
> Belgian Biodiversity Platform (http://www.biodiversity.be)
> PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
> No trees were killed in the creation of this message.
> However, many electrons were terribly inconvenienced.



-- 
Julien Cigar
Belgian Biodiversity Platform (http://www.biodiversity.be)
PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20170831/ec2a2ec2/attachment.sig>