Very low disk performance on 5.x

[Bakul Shah]

It may make sense to look at measured bandwidth as a
percentage of *guaranteed not to exceed* bandwidth of the
disk setup -- what is the theoretical max bandwidth writing
to a raw partition (and assuming zero cpu overhead, latency,
seek time)?  This will help in figuring out how to maximize
end-to-end performance, taking into account filesystem
overhead, data integrity tradeoffs etc.  One has to measure
each component to find and fix the top N bottlenecks.  This
sort of critical analysis is what allowed people to make
TCP/IP blazingly fast.

For instance, Hartland reported max read/write performance of
about 260M/s.  *If* his disks can do, say, 80MB/s, for a 5
disk RAID5 he'd get 320MB/s.  If so, his end-to-end measured
performance would be over 80% of max (for whatever data
integrity guarantees).  Not too shabby:-)

Improving end-to-end disk/os/filesystem performance can be
quite an exciting (and time consuming) project.
- do we have hooks to measure performance of each component?
- do we have tests for these?
- do we know the top 3 bottlenecks?
- do we have a framework for independently experimenting
  with each component?  (geom definitely helps here!)
- do we have a framework for experimenting with various
  filesystem layout schemes?  with various caching
  strategies?  with various allocation strategies?
- tools for locating bottlenecks due to mistuning?
- tools to help tune the system?
- how about tools to auto-tune the system?:)

I agree with the questions you raised, but it seems we need
to move the discussion to a meta level!  I fully realize that
in a volunteer project a) there are never enough people b)
people works on what they like....  May be someone like you
can inspire & put together a team to make freebsd the best
damn storage OS in the world!  Actually what would be really
neat is to factor out all FS code across all *BSDs so that
rather than fixing the same old bugs N times, people work on
different FS designs.

The reality is that open OSes are still nowhere near what SGI
boxes of a few years ago could do.  I have heard of some NAS
box vendors achieving really good performance but that
doesn't help us in the open OS community (not to mention
their advances will be lost when the company dies or is
garbage collected by the likes of EMC, netApp).