Strange performance characteristics with ZFS

[Steven Schlansker]

Hello everyone,
I'm experiencing what (to me) seems to be strange performance with  
ZFS.  I have it set up as follows:

[steven at universe /universe]$ sudo zpool status
   pool: universe
state: ONLINE
scrub: resilver completed with 0 errors on Mon Jul 16 10:33:52 2007
config:

         NAME        STATE     READ WRITE CKSUM
         universe    ONLINE       0     0     0
           raidz1    ONLINE       0     0     0
             ad10    ONLINE       0     0     0
             ad6     ONLINE       0     0     0
             ad4     ONLINE       0     0     0
           raidz1    ONLINE       0     0     0
             ad12    ONLINE       0     0     0
             ad16    ONLINE       0     0     0
             ad14    ONLINE       0     0     0

I share it over NFS to a linux machine.  I'm trying to write a very  
large (30GB) disk image to it over nfs, and it goes in short fits and  
bursts.

It copies a few (mega?)bytes of data (not sure how much exactly), and  
top shows:
   605 root        1   4    0  3616K  1068K -        1:17  5.18% nfsd
   603 root        1   4    0  3616K  1068K -        1:33  4.69% nfsd
   602 root        1   4    0  3616K  1068K -        1:09  4.35% nfsd
   604 root        1   4    0  3616K  1068K -        1:24  3.66% nfsd

Then, the copy on the linux box freezes entirely and top shows:
   603 root        1  -4    0  3616K  1068K zfs:(&   1:34  4.98% nfsd
   605 root        1  -4    0  3616K  1068K zfs      1:17  4.25% nfsd
   604 root        1  -4    0  3616K  1068K zfs      1:24  4.05% nfsd
   602 root        1  -4    0  3616K  1068K zfs      1:10  3.96% nfsd

While the nfsds are stuck in the 'zfs' state, ssh is very  
unresponsive (few seconds per keystroke)
After a few seconds of being hung like this, the copy resumes and the  
nfsds go back to a blank state.

I guess the ssh hanging is the most annoying symptom, as it makes it  
hard to use the system while copying.  It would be nice to speed the  
transfers up too though.

Is this expected behavior?  Is there some way I can make it run  
faster/better?

[steven at universe /universe]$ uname -a
FreeBSD universe.stevenschlansker.is-a-geek.org 7.0-CURRENT FreeBSD  
7.0-CURRENT #1: Thu Jul 12 11:48:11 PDT 2007      
root at universe.stevenschlansker.is-a-geek.org:/usr/obj/usr/src/sys/ 
GENERIC  amd64


Thanks,
Steven