Fun with HAST and inter-host connections

[Julien Cigar]

On Wed, Aug 30, 2017 at 10:40:03PM +0100, Frank Leonhardt wrote:
> Please note - I'm pushing what can be done with commodity hardware for 
> amusement here, and trying to do interesting things with FreeBSD. I *do* 
> want to do it this way.
> 
> Okay, so the game is this: I've got some identical hosts with disks and 
> I've been working on ways of clustering them for a long time. It's what 
> I do for fun, right?
> 
> I'm currently playing with failover storage. Not necessarily with HAST; 
> iSCSI is fun and I'm messing with geom stuff in general. But let's stick 
> with HAST as it illustrates the dilemma.
> 
> When I started this game, 1Gb Ethernet was blistering. Now it's not so 
> hot. How do I "network" the hosts with as much throughput as possible 
> (with IP sockets, preferably)? Options:
> 
> 1) 10Gb Ethernet is expensive. 10Gb switches even more so, but I can do 
> point-to-point.
> 
> 2) LAG is more about failover than speed. And anyway, you end up needing 
> a lot of Ethernet ports on each host and it soon gets crazy.
> 
> 3) Fibrechannel - if I bought a few old (cheap) fiberchannel cards, I'm 
> not sure how I could use them point-to-point. Is this possible and has 
> anyone done it?
> 
> 4) USB 3.1. 10Gb. PCIe cards cost about $30. Now this sounds fun. IP 
> over USB anyone?
> 
> Now please don't advise me to get a pair of fabric switches and do the 
> job properly. That's not what this is about. I want to see if it's 
> possible to make a fast(er) storage solution using cheap components. A 
> sort of Redundant Array of Inexpensive NAS.
> 
> Thoughts anyone? In particular, is the USB 3.1 idea crazy? And is anyone 
> else crazy enough to be trying the same thing?

I've set up a low-cost redundant storage here at work with ZFS + iSCSI,
basically a zpool mirror over 2 local disks and 2 iSCSI disks. Failover
is done through CARP and devd. It works well (except when Jumbo Frames
are turned on) for almost a year now. Just for extra security, and also
because some people (smarter than me in this area) advised me against 
this approach, I've setup a third node with ZFS replication (zrep).
Basically if you can afford to lose "some" data, I would suggest to go
with ZFS replication.

> 
> Thanks, Frank.
> 
> 
> _______________________________________________
> freebsd-questions at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-questions
> To unsubscribe, send any mail to "freebsd-questions-unsubscribe at freebsd.org"

-- 
Julien Cigar
Belgian Biodiversity Platform (http://www.biodiversity.be)
PGP fingerprint: EEF9 F697 4B68 D275 7B11  6A25 B2BB 3710 A204 23C0
No trees were killed in the creation of this message.
However, many electrons were terribly inconvenienced.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-questions/attachments/20170831/fbb68d75/attachment.sig>