ZFS and Wired memory, again

[Anton Sayetsky]

Hello,

I'm planning to deploy a ~150 TiB ZFS pool and when playing with ZFS
noticed that amount of wired memory is MUCH bigger than ARC size (in
absence of other hungry memory consumers, of course). I'm afraid that
this strange behavior may become even worse on a machine with big pool
and some hundreds gibibytes of RAM.

So let me explain what happened.

Immediately after booting system top says the following:
=====
Mem: 14M Active, 13M Inact, 117M Wired, 2947M Free
ARC: 24M Total, 5360K MFU, 18M MRU, 16K Anon, 328K Header, 1096K Other
=====
Ok, wired mem - arc = 92 MiB

Then I started to read pool (tar cpf /dev/null /).
Memory usage when ARC size is ~1GiB
=====
Mem: 16M Active, 15M Inact, 1410M Wired, 1649M Free
ARC: 1114M Total, 29M MFU, 972M MRU, 21K Anon, 18M Header, 95M Other
=====
1410-1114=296 MiB

Memory usage when ARC size reaches it's maximum of 2 GiB
=====
Mem: 16M Active, 16M Inact, 2523M Wired, 536M Free
ARC: 2067M Total, 3255K MFU, 1821M MRU, 35K Anon, 38M Header, 204M Other
=====
2523-2067=456 MiB

Memory usage a few minutes later
=====
Mem: 10M Active, 27M Inact, 2721M Wired, 333M Free
ARC: 2002M Total, 22M MFU, 1655M MRU, 21K Anon, 36M Header, 289M Other
=====
2721-2002=719 MiB

So why the wired ram on a machine with only minimal amount of services
has grown from 92 to 719 MiB? Sometimes I can even see about a gig!
I'm using 9.2-RELEASE-p1 amd64. Test machine has a T5450 C2D CPU and 4
G RAM (actual available amount is 3 G). ZFS pool is configured on a
GPT partition of a single 1 TB HDD.
Disabling/enabling prefetch does't helps. Limiting ARC to 1 gig doesn't helps.
When reading a pool, evict skips can increment very fast and sometimes
arc metadata exceeds limit (2x-5x).

I've attached logs with system configuration, outputs from top, ps,
zfs-stats and vmstat.
conf.log = system configuration, also uploaded to http://pastebin.com/NYBcJPeT
top_ps_zfs-stats_vmstat_afterboot = memory stats immediately after
booting system, http://pastebin.com/mudmEyG5
top_ps_zfs-stats_vmstat_1g-arc = after ARC grown to 1 gig,
http://pastebin.com/4AC8dn5C
top_ps_zfs-stats_vmstat_fullmem = when ARC reached limit of 2 gigs,
http://pastebin.com/bx7svEP0
top_ps_zfs-stats_vmstat_fullmem_2 = few minutes later,
http://pastebin.com/qYWFaNeA

What should I do next?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: logs.txz
Type: application/octet-stream
Size: 14892 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20131122/f118859d/attachment.obj>