kern/169480: ZFS stalls on heavy I/O

[Levent Serinol]

>Number:         169480
>Category:       kern
>Synopsis:       ZFS stalls on heavy I/O
>Confidential:   no
>Severity:       serious
>Priority:       low
>Responsible:    freebsd-bugs
>State:          open
>Quarter:        
>Keywords:       
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Wed Jun 27 09:20:07 UTC 2012
>Closed-Date:
>Last-Modified:
>Originator:     Levent Serinol
>Release:        9.0 and 10.0
>Organization:
>Environment:
FreeBSD localhost 9.0-RELEASE FreeBSD 9.0-RELEASE #0: Wed Mar 21 11:15:41 EET 2012     root at webmail122.mynet.com:/usr/obj/usr/src/sys/GENERIC  amd64
>Description:
 Under heavy I/O load we face freeze problems on ZFS volumes on both Freebsd 9 Release and 10 Current versions. Machines are HP servers (64bit) with HP Smart array 6400 raid controllers (with battery units). Every da device is a hardware raid5 where each one includes 9x300GB 10K SCSI hard drivers. Main of I/O pattern happens on local system except some small NFS I/O from some other servers (NFS lookup/getattr/ etc.). These servers are mail servers (qmail) with small I/O patterns (64K Read/Write).  Below you can  find procstat output on freeze time. write_limit is set to 200MB because of the huge amount of txg_wait_opens observed before. Every process stops on D state I think due to txg queue and other 2 queues are full. Is there any suggestion to fix the problem ?

btw inject_compress is the main process injecting emails to user inboxes (databases). Also, those machines were running without problems on Linux/XFS filesystem. For a while ago, we started migration from Linux to Freebsd

http://pastebin.com/raw.php?i=ic3YepWQ
>How-To-Repeat:

>Fix:


>Release-Note:
>Audit-Trail:
>Unformatted: