24 TB UFS2 reality check ?

[Jeff Mohler]

> Two final questions:
>
> 1. What would _you_ do with 24 1 TB disks and a 24 port 3ware card ?  Assume an i386, 4 GB machine, and that fsck is workable because of "newfs -i 131072"
---
Im biased.  If I had to have 24TB online, id get a netapp.  24TB of
data has gotta be worth some real money, so I'd spend it.  No..youre
not wrong for NOT doing that, but, I wouldnt consider the 24TB
nightmare myself, ive been there before.  I never full healed from
that mess.

But if I had to steer you in a specific direction, plan for total
failure.  HDD's are built to do one thing.  Fail.

They occasionally hold data, but in reality, theyre just built to
fail.  Some are designed to fail sooner than others.  You have 24 of
them racing for failure.

Sounds pessimistic, but, good backup & DR strategies are built around this.


> 2. What number should I ask my vendor (3ware) to do the rebuild calculations ?  You are talking about IOPS/s - I think I should ask them how many IOPS/s the card does when rebuilding a 24 disk raid-6 array, and then combine that with the IOPS/s I see in my normal workload.
---
Rebuild calculations are based around how fast the drives are, cheap
SATA is about 14ms track to track (longer full seek) and its highly
random, and it has to compete with user/system workload.  Thats just
not possible to state, too many variables.  IOPS are not a card issue,
its a physical drive issue.  24TB of FCAL would rebuild faster than
SATA, for example.  The intelligence of the card itself could come
into play, in case it is able to use command queuing to the
drives/etc...and if the SATA drives fully support it as well.
Depending what you want the system to do for users during the rebuild,
prioritize the card appropriately.

> How do you measure IOPS/s in FreeBSD on a running machine ?
---
iostat -x  is a pretty good way to measure that, for the most part.
Im prepared to hear about different/better ways.