Fun with HAST and inter-host connections

[Frank Leonhardt]

Please note - I'm pushing what can be done with commodity hardware for 
amusement here, and trying to do interesting things with FreeBSD. I *do* 
want to do it this way.

Okay, so the game is this: I've got some identical hosts with disks and 
I've been working on ways of clustering them for a long time. It's what 
I do for fun, right?

I'm currently playing with failover storage. Not necessarily with HAST; 
iSCSI is fun and I'm messing with geom stuff in general. But let's stick 
with HAST as it illustrates the dilemma.

When I started this game, 1Gb Ethernet was blistering. Now it's not so 
hot. How do I "network" the hosts with as much throughput as possible 
(with IP sockets, preferably)? Options:

1) 10Gb Ethernet is expensive. 10Gb switches even more so, but I can do 
point-to-point.

2) LAG is more about failover than speed. And anyway, you end up needing 
a lot of Ethernet ports on each host and it soon gets crazy.

3) Fibrechannel - if I bought a few old (cheap) fiberchannel cards, I'm 
not sure how I could use them point-to-point. Is this possible and has 
anyone done it?

4) USB 3.1. 10Gb. PCIe cards cost about $30. Now this sounds fun. IP 
over USB anyone?

Now please don't advise me to get a pair of fabric switches and do the 
job properly. That's not what this is about. I want to see if it's 
possible to make a fast(er) storage solution using cheap components. A 
sort of Redundant Array of Inexpensive NAS.

Thoughts anyone? In particular, is the USB 3.1 idea crazy? And is anyone 
else crazy enough to be trying the same thing?

Thanks, Frank.