ZFS pool with a large number of filesystems

[PK1048]

> On Apr 4, 2016, at 20:38, Wim Lewis <wiml at omnigroup.com> wrote:
> 
> I'm curious how many ZFS filesystems are reasonable to have on a single machine (in a single zpool). We're contemplating a design in which we'd have tens of thousands, perhaps a couple hundred thousand, filesystems mounted out of the same pool. Before we go too far into investigating this idea: Does anyone have real-world experience doing something like that? Is it a situation that ZFS-on-FreeBSD is engineered to handle with good performance? Is there a rough estimate of the resources consumed per additional filesystem (in terms of kernel VM and disk space)?
> 
> Thanks for any insight or advice (even, or especially, if the answer is "that's crazy, don't do that" :) )

In a previous life I ran ZFS on Solaris (10) with hundreds of datasets, but all off of the top level zpool. We also had hundreds of thousands of snapshots. There was no noticeable hit to user performance, but as others have pointed out, getting a listing of all the datasets and snapshots was not a fast operation. zfs list -t filesystem on the other hand was fast.

Fast forwarding to today… I have a handful of FreeBSD boxes running ZFS with less than 100 datasets but with _hourly_ snapshots kept for 30 days. That is 720 snapshots per dataset. Our largest server has about 50 datasets, so about 36,000 snapshots. zfs list -t filesystem is fast while zfs list -t snapshot is slow, but I don’t do that very often. No performance hit to normal I/O.

The FreeBSD hardware is Xeon X55xx or E55xx, usually 2 quad core CPUs, so 8 or 16 vCPU. RAM varies from  low of 12GB to a high of 80GB. Systems are running a mix of vBox VMs, file service, backup service (including Mac OS Timemachine backups via NetATalk).