Options for synchronising filesystems

[Isaac Levy]

Hi Brian, All,

This email has one theme: GEOM! :)

On Sep 24, 2005, at 10:10 AM, Brian Candler wrote:

> Hello,
>
> I was wondering if anyone would care to share their experiences in
> synchronising filesystems across a number of nodes in a cluster. I  
> can think
> of a number of options, but before changing what I'm doing at the  
> moment I'd
> like to see if anyone has good experiences with any of the others.
>
> The application: a clustered webserver. The users' CGIs run in a  
> chroot
> environment, and these clearly need to be identical (otherwise a  
> CGI running
> on one box would behave differently when running on a different box).
> Ultimately I'd like to synchronise the host OS on each server too.
>
> Note that this is a single-master, multiple-slave type of filesystem
> synchronisation I'm interested in.

I just wanted to throw out some quick thoughts on a totally different  
approach which nobody has really explored in this thread, solutions  
which are production level software. (Sorry if I'm repeating things  
or giving out info yall' already know:)

--
Geom:
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom- 
intro.html

The core Disk IO framework for FreeBSD, as of 5.x, led by PHK:
http://www.bsdcan.org/2004/papers/geom.pdf

This framework itself is not as useful to you as the utilities which  
make use of it,

--
Geom Gate:
http://kerneltrap.org/news/freebsd?from=20

Network device-level client/server disk mapping tool.
(VERY IMPORTANT COMPONENT, it's reportedly faster, and more stable  
than NFS has ever been- so people have immediately and happily  
deployed it in production systems!)

--
Gvinum and Gmirror:

Gmirror
http://people.freebsd.org/~rse/mirror/
http://www.ie.freebsd.org/doc/en_US.ISO8859-1/books/handbook/geom.html

(Sidenote: even Greg Lehey (original author of Vinum), has stated  
that it's better to use Geom-based tools than Vinum for the  
forseeable future.)

--
In a nutshell, to address your needs, let me toss out the following  
example setup:

I know of one web-shop in Canada, which is running 2 machines for  
every virtual cluster, in the following configuration:

2 servers,
4 SATA drives per box,
quad copper/ethernet gigabit nic on each box

each drive is mirrored using gmirror, over each of the gigabit  
ethernet nics
each box is running Vinum Raid5 across the 4  mirrored drives

The drives are then sliced appropriately, and server resources are  
distributed across the boxes- with various slices mounted on each box.
The folks I speak of simply have a suite of failover shell scripts  
prepared, in the event of a machine experiencing total hardware failure.

Pretty tough stuff, very high-performance, and CHEAP.

--
With that, I'm working towards similar setups, oriented around  
redundant jailed systems, with an eventual end to tie CARP (from pf)  
into the mix to make for nearly-instantaneous jailed failover  
redundancy- (but it's going to be some time before I have what I want  
worked out for production on my own).

Regardless, it's worth tapping into the GEOM dialogues, as there are  
many new ways of working with disks coming into existence- and the  
GEOM framework itself provides an EXTREMELY solid base to bring  
'exotic' disk configurations up to production level quickly.
(Also noteworthy, there's a couple of encrypted disk systems based on  
GEOM emerging now too...)

--
Hope all that helps,

Best,
.ike