HPC and zfs.

[Ronald Klop]

On Mon, 06 Feb 2012 16:52:11 +0100, Peter Ankerstål <peter at pean.org> wrote:

> Hi,
>
> I want to investigate if it is possible to create your own usable HPC  
> storage using zfs and some
> network filesystem like nfs.
>
> Just a thought experiment..
> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more)
> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD  
> deives for  cache.
> Preferrably in  mirror where applicable.
>
> Connected to this machine we will have about 410 3TB drives to give  
> approx
> 1PB of usable storage in a 8+2 raidz configuration.
>
> Connected to this will be a ~800 nodes big HPC cluster that will access  
> the storage in parallell
> is this even possible or do we need to distribute the meta data load  
> over many servers? If that is the case,
> does it exist any software for FreeBSD that could  accomplish this  
> distribution (pNFS  dosent seem to be
> anywhere close to usable in FreeBSD) or do I need to call NetApp or  
> Panasas right away? It would be
> really nice if I could build my own storage solution.
>
> Other possible solutions to this problem is extremley welcome.
>
> Best Regards
> Peter Ankerstål

You might make a call to Backblaze.

http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

Ronald.