RELENG_7 heavy disk = system crawls

[grarpamp]

Hi. I'm running RELENG_7 on an older single P4. It has a lot of
disk on it that does mainly sequential read/write of gigs of data.
The data disks are hanging off a dumb ata133 pdc20269 card, they
use geli aes 128 and zfs sha256, single spindles. Free ram, no swap,
free disk, no net, etc.

In short, whenever I'm doing sequential disk stuff, human interface
system performance tanks, big time.  Keystrokes take a second or
two to appear, X11 focus raises take forever, firefox is a dog...
after waiting 10min for the system to load it off disk, run it and
paint it to X11, mouse movements are largely ignored unless moved
very slowly. Even typing killall <dogs> is painful to do :)

I am very near the limit on wired mem, despite telling zfs to use
only 96MiB. And sometimes X11 can't get the ram it wants and dies
mid session. But neither of those should affect performance, the
kernel will deal with low mem one way or another, drastically, not
by bogging down [no swap enabled anyways] right?

I used to do similar stuff on RELENG_4 on an old dual PII with plain
UFS and I could run gigs of disk data, all the spindles, 8 of them,
through sha1, all at once, and still have good interactive performance.

Yet fire off one sequential read/write, perhaps piped into sha1 for
more cpu load and it's game over. I can renice the user process way
down into idprio with no effect. So I doubt giving rtprio to all the
human processes would do anything.

Is RELENG_7 or GELI+ZFS the dog here? Top of course shows
GELI+ZFS eating all the cpu. Ok, fine, so how do I say, hey, I don't
care about disk, it'll finish eventually, just give me my gui and
keystrokes back?

It just doesn't seem right that this one subsystem should be
monopolizing the cpu this way? Help/cluebat? Thanks.