ZFS: arc_meta consumes *all* ram

[Ian Downes]

Is vfs.zfs.arc_meta_limit supposed to be a (relatively) hard limit on
cached metadata?

I've limited the arc size with arc_max but how do I effectively limit
the caching of meta data?

Suggestions appreciated!

details:
ZFS is exceeding vfs.zfs.arc_meta_limit on some of my boxes; consuming
all available RAM, paging everything out and bringing the system to its
knees.

$ uname -a
FreeBSD local 8.2-RELEASE FreeBSD 8.2-RELEASE #0: Fri Jul  8 00:54:56 UTC 2011     root at 8.8.8.8:/usr/obj/usr/src/sys/XENHVM  amd64
$ sysctl vfs.zfs | grep arc_meta
vfs.zfs.arc_meta_limit: 1610612736
vfs.zfs.arc_meta_used: 12183379056

Note that this is 7-8X over arc_meta_limit and was all the available
RAM on the box.

This can be reproduced on several boxes (8.2-RELEASE patched to ZFS
5/28 and 9.0-RELEASE) when periodic/security/100.chksetuid runs and does
a find over all filesystems.