RELENG_7 heavy disk = system crawls

[grarpamp]

> nice only affects userland

Well, you can set {id,rt}prio and nice on kernel processes. Then
look at top to see the nice column change. Have no idea what effect
it has nor what the non '-' chars on those procs in that column
mean.

> Do you *need* geli+zfs?

Encryption = required.
ZFS... well I like the checksum all the way back to the uberblock
feature, raidz2, ditto blocks, compression and the admin model.
geom offers encryption, single block checksum authority and raid1/3.

> hardware crypto accellerators

The soekris ones work and are cheap. I thougt I saw posts that show
openssl -speed on today's fast cpu's being faster than the accel
cards. Disk crypto is symmetric, not initial pki session setup.

And with ~40MB/s of encryption, while building world no less, cpu
may not be the issue. So long as I'm not hitting those geli+zfs
disks, things are smooth. I want to try the system with geli+ufs2
sometime.

type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128 cbc      36555.26k    38074.95k    38999.93k    39258.38k    39176.80k

> throw lots of cpu (e.g. phenom2 x4) at it

That would only help it get done busying out the system sooner, not
balance things out while actively under load. Which as a file server,
it always is.

Well, ok, it will help after the system is able to do spindle ->
geli -> fs -> process at the max sustained read/write speed of the
spindles. Which is about 56MiB/sec reading in this case. Which is
over 10x faster than I'm getting now. Which means I'd need maybe
10 x 1.8GHz worth of cpu before I have any free cycles to devote
to the user interface :)

Maybe I'm just clueless this month and the list is too busy to beat
me about the head with it.

> The nForce pata controller doesn't list an irq, seems odd?
> device 6.0 on pci0

This one doesn't either:
atapci1: <Intel ICH4 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 31.1 on
pci0

boot -v and it appears in the irq routing stuff around that line.
Then vmstat -i and systat -vmstat 1 also give some clues when device
is dd'd. Onboard PATA is always irq14/15 as I've seen.