ZFS stalls on Heavy I/O

[Dean Jones]

On Wed, Jun 27, 2012 at 2:15 AM, Levent Serinol <lserinol at gmail.com> wrote:
> Hi,
>
>  Under heavy I/O load we face freeze problems on ZFS volumes on both
> Freebsd 9 Release and 10 Current versions. Machines are HP servers (64bit)
> with HP Smart array 6400 raid controllers (with battery units). Every da
> device is a hardware raid5 where each one includes 9x300GB 10K SCSI hard
> drivers. Main of I/O pattern happens on local system except some small NFS
> I/O from some other servers (NFS lookup/getattr/ etc.). These servers are
> mail servers (qmail) with small I/O patterns (64K Read/Write).  Below you
> can  find procstat output on freeze time. write_limit is set to 200MB
> because of the huge amount of txg_wait_opens observed before. Every process
> stops on D state I think due to txg queue and other 2 queues are full. Is
> there any suggestion to fix the problem ?
>
> btw inject_compress is the main process injecting emails to user inboxes
> (databases). Also, those machines were running without problems on
> Linux/XFS filesystem. For a while ago, we started  migration from Linux to
> Freebsd
>
>
> http://pastebin.com/raw.php?i=ic3YepWQ
> _______________________________________________

Looks like you are running dedup with only 12 gigs of ram?

Dedup is very ram hungry and the dedup tables are probably no longer
fitting entirely in memory and therefore the system is swapping and
thrashing about during writes.

Also ZFS really prefers to directly address drives instead of RAID
controllers.  It can not guarantee or know what the controller is
doing behind the scenes.