Multi node storage, ZFS

[Michal]

On 24/03/2010 16:20, Freddie Cash wrote:
Horribly, horribly, horribly complex.  But, then, that's the Linux world.
>  :)

Yes I know, it's not very clean, but was trying to gather ideas and I
found that

> 
> Server 1:  bunch of disks exported via iSCSI
> Server 2:  bunch of disks exported via iSCSI
> Server 3:  bunch of disks exported via iSCSI
> 
> "SAN" box:  uses all those iSCSI exports to create a ZFS pool
> 
> Use 1 iSCSI export from each server to create a raidz vdev.  Or multiple
> mirror vdevs.  When you need more storage, just add another server full of
> disks, export them via iSCSI to the "SAN" box, and expand the ZFS pool.
> 
> And, if you need fail-over, on your "SAN" box, you can use HAST at the lower
> layers (currently only available in 9-CURRENT) to mirror the storage across
> two systems, and use CARP to provide a single IP for the two boxes.
> 
> ---------------------------------------------------------------------

This is pretty much what I have been looking for, I don't mind using a
SAN Controller server in which to deal with all of this in fact I
expected that, but I wanted to present the disks from a server full of
HDD's (which in effect is just a storage device) and then join them up.
I've briefly looked over RAIDz, will give it a good reading over later.
I'm thinking 6 disks in each server, and two raidz vdev created from 3
disks in each server. I can them serve them to the network. I've never
used ISCSI on FreeBSD however, I played with AOE on different *nix's so
I will give ISCSI a good looking over.

> Yes, you save space, but your throughput will be horribly horribly horribly
> low.  RAID arrays should be narrow (1-9 disks), not wide (30+ disks), and
> then combined into a larger array (multiple small RAID6 arrays joined into a
> RAID0 stripe).

Oh Yes I agree, was doing some very crude calculations and the
difference in space was quite a lot, but no I would never do that in
reality

> If you were to do something like this, I'd make sure to have a fast
>local ZIL (log) device on the head node.  That would reduce latency
>for writes, you might also do the same for reads.  Then your bulk
>storage comes from the iSCSI boxes.
>
>Just a thought.

I've not come across ZIL so I think I will have to do my research


>At least in theory you could use geom_gate and zfs I suppose, never
>tried it though.
>ggatec(8), ggated(8) are your friends for that.
>
>Vince

Just had a look at ggatec, I've not seen or heard of that so I will
continue looking through that.


Many thanks to all, if I get something solid working I will be sure to
update the list with what will hopefully be a very cheap (other then
HDD's) SAN