ZFS eats up 3GB of ram then the machine starts dropping connections

[Peter Ankerstål]

Hi,

I have a machine with a 1.3T pool, hosting about 10 jails.. When the  
periodic daily was running
ZFS began to eat memory, up to 3GB (out of the installed 4GB).   
Something like 200MB Inuse, 3GB Wired
and just a few MB Free. After that the machine was getting really  
laggy and began dropping connections.
httpd was giving half web-pages and sshd was shutting down the  
connections with some error message
"Invalid package size" or "Invalid MAC"

I tried to look it up and found that vfs.zfs.prefetch_disable="1"  
could help, but problem remains. It seems like
zfs takes all the memory it could possibly get and then refuses to  
give it back. I've tried different approaches
to replicate this porblem and something like ls -R / would do exactly  
the same thing. It starts eating ram shitloads
of ram, and when I stop... I also stops eating ram but the amout Wired  
ram stays there. Doesnt go up or (down)
until you do another ls or find.


What about the machine then:

# uname -a
FreeBSD ninja.jails.se 7.0-RELEASE FreeBSD 7.0-RELEASE #0: Sun Feb 24  
10:35:36 UTC 2008     root at driscoll.cse.buffalo.edu:/usr/obj/usr/src/ 
sys/GENERIC  amd64

# cat /boot/loader.conf
vfs.zfs.prefetch_disable="1"
vm.kmem_size_max="1073741824"
vm.kmem_size="1073741824"

# cat /etc/sysctl.conf
kern.maxvnodes=400000


Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Core(TM)2 Quad CPU    Q6600  @ 2.40GHz (2401.93-MHz K8- 
class CPU)
Origin = "GenuineIntel"  Id = 0x6fb  Stepping = 11
Features 
= 
0xbfebfbff 
< 
FPU 
,VME 
,DE 
,PSE 
,TSC 
,MSR 
,PAE 
,MCE 
,CX8 
,APIC 
,SEP 
,MTRR 
,PGE 
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4
usable memory = 3744874496 (3571 MB)
avail memory  = 3590160384 (3423 MB)
ACPI APIC Table: <MSTEST OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
cpu2 (AP): APIC ID:  2
cpu3 (AP): APIC ID:  3