HPC and zfs.

Jeremy Chadwick freebsd at jdc.parodius.com
Mon Feb 6 16:22:08 UTC 2012


On Mon, Feb 06, 2012 at 04:52:11PM +0100, Peter Ankerst?l wrote:
> I want to investigate if it is possible to create your own usable
> HPC storage using zfs and some
> network filesystem like nfs.
> 
> Just a thought experiment..
> A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more)
> I addition the machine will use 3-6 SSD drives for ZIL and 3-6 SSD
> deives for  cache.
> Preferrably in  mirror where applicable.
> 
> Connected to this machine we will have about 410 3TB drives to give approx
> 1PB of usable storage in a 8+2 raidz configuration.
> 
> Connected to this will be a ~800 nodes big HPC cluster that will
> access the storage in parallell
> is this even possible or do we need to distribute the meta data load
> over many servers? If that is the case,
> does it exist any software for FreeBSD that could  accomplish this
> distribution (pNFS  dosent seem to be
> anywhere close to usable in FreeBSD) or do I need to call NetApp or
> Panasas right away? It would be
> really nice if I could build my own storage solution.
> 
> Other possible solutions to this problem is extremley welcome.

For starters I'd love to know:

- What single motherboard supports up to 192GB of RAM
- How you plan on getting roughly 410 hard disks (or 422 assuming
  an additional 12 SSDs) hooked up to a single machine

If you are considering investing the time and especially money (the cost
here is almost unfathomable, IMO) into this, I strongly recommend you
consider an actual hardware filer (e.g. NetApp).  Your performance and
reliability will be much greater, plus you will get overall better
support from NetApp in the case something goes wrong.  In the case you
run into problems with FreeBSD (and I can assure you in this kind of
setup you will) with this kind of extensive setup, you will be at the
mercy of developers' time/schedules with absolutely no guarantee that
your problem will be solved.  You definitely want a support contract.
Thus, go NetApp.

-- 
| Jeremy Chadwick                                 jdc at parodius.com |
| Parodius Networking                     http://www.parodius.com/ |
| UNIX Systems Administrator                 Mountain View, CA, US |
| Making life hard for others since 1977.             PGP 4BD6C0CB |



More information about the freebsd-fs mailing list