20TB Storage System (fsck????)

[David Schultz]

On Thu, Sep 04, 2003, Andrew Kinney wrote:
> Our experience has been that with 4GB of RAM (or more) you 
> really must increase your KVA to 2GB, leaving only 2GB of UVA.  
> So, I would concur with what Julian said.
> 
> <ducks his head to avoid the rotten tomatoes that are sure to be 
> thrown> ;-)
> 
> With the lack of third party filesystem support in FreeBSD, might 
> you be better served by looking at a Linux system running 
> ReiserFS or one of the other file systems designed for such 
> behemoth disk systems?

I understand that ReiserFS fares rather poorly on large
filesystems.  People have complained about it overflowing the
kernel heap when they try to create massive volumes.  Perhaps more
recent versions have addressed this, and I haven't verified this
myself.

> These days, I think Sun even gives away Solaris licenses with their 
> low end x86 servers, so that might even be an option.

Umm...the Sun UFS implementation doesn't support >2TB volumes,
since it uses 32-bit block numbers.

> UFS is great, but there are other filesystems out there that have 
> already addressed such problems from their use in academic, 
> government, and scientific computing where gigantic filesystems 
> tend to be more prevalent.

Yep.  The most significant problem I see with UFS is that
recovering from a crash or power failure is an O(data) operation,
and that just doesn't scale well.  Other designs (the most
prevalent of which involve logging) support O(1) recovery.

> >From my brief research on the subject, the FreeBSD community 
> has been highly resistant to supporting third party filesystems 
> precisely because nobody with such needs as yours has ever 
> contributed the code necessary to make third party filesystem 
> support a reality.  The response is usually something like "if you 
> want it, go program it and submit it to a committer for review," 
> which is really the only reasonable response that can be made 
> given FreeBSD's goals.

There's probably only a handful of committers who have access to
the quantities of storage that would make this kind of filesystem
development interesting.  Meanwhile, UFS2 is perfectly adequate
for most systems.  I would imagine that an interested company
would have to step forward and fund this sort of work.  Moreover,
there are licensing issues that would have to be considered to do
a port of an existing filesystem...