tuning FFS for large files Re: A specific example of a disk i/o problem

[Dieter]

I found a clue!  The problem occurs with my big data partitions,
which are newfs-ed with options intended to improve things.

Reading a large file from the normal ad4s5b partition only delays other
commands slightly, as expected.  Reading a large file from the tuned
ad4s11 partition yields the delay of minutes for other i/o.

# tunefs -p /dev/ad4s5b
tunefs: ACLs: (-a)                                         disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: gjournal: (-J)                                     disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  1024
tunefs: average file size: (-f)                            16384
tunefs: average number of files in a directory: (-s)       64
tunefs: minimum percentage of free space: (-m)             8%
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)                                 

# tunefs -p /dev/ad4s11
tunefs: ACLs: (-a)                                         disabled
tunefs: MAC multilabel: (-l)                               disabled
tunefs: soft updates: (-n)                                 enabled
tunefs: gjournal: (-J)                                     disabled
tunefs: maximum blocks per file in a cylinder group: (-e)  57984
tunefs: average file size: (-f)                            67108864
tunefs: average number of files in a directory: (-s)       16
tunefs: minimum percentage of free space: (-m)             8%
tunefs: optimization preference: (-o)                      time
tunefs: volume label: (-L)                     

Here is the newfs command used for creating large data partitions:
newfs -e 57984 -b 65536 -f 8192 -g 67108864 -h 16 -i 67108864 -U -o time $partition

Even this isn't tuned the way I wanted to.
-g * -h must be less than 4 G due to 32 bit problem (system panics).
Note the 32 bit problem is in the filesystem code, I'm running amd64.
IIRC there is a PR about this.  (I'm assuming the bug hasn't been fixed yet)
Result is that I must specify -g and -h smaller than they should be.
And they have way more inodes than needed. (IIRC it doesn't actually
use -i 67108864)

Partition sizes range from ~200 GB to 1.5 TB.  A small number of
directories, perhaps a dozen.  About 1/2 the files are small, a few KB,
the other half are large, 500MB-25GB.

Goals are to minimize seeking and use space efficiently.  Files are
written sequentially, read mostly (99.99%) sequentially, no editing.

I still think it may have something to do with a common resource
such as disk cache, as it can affect other disks, including swap.
Hmmm, does swap use the disk cache?  But now it appears that FFS
tuning is the cause of the bottleneck.

Do these tuning parameters increase the number of some data structure
needed?  Can I crank that up?  Do I have to change the FFS tuning?
Suggestions?