ZFS - poor performance with "large" directories

[Albert Cervin]

Hi all,

Please feel free to direct me to a list that is more suitable.

We are trying to set up a fileserver solution for a web application that we
are building. This fileserver is running FreeBSD 10.2 and ZFS. Files are
written over CIFS with Samba running on the fileserver host.

However, we are seeing en exponential decrease in performance to write to
the file server when the number of files in the directory grows (when it
goes up to ~6000 files it becomes unusable and the write time has gone from
a fraction of a second to ten seconds).

We ran the same setup on a Linux machine with an ext4 file system which did
NOT suffer from this performance degradation.

Our first reaction was to remove Samba from the equation. I ran a test
where i tried to copy a folder with a large amount of files and then ran a
test with the same folder as a zip.

So,

cp -r folder_with_lots_of_files copy_of_folder_with_lots_of_files

gives an iostat output that looks like this for the zpool (zpool iostat
frosting 1):

pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
frosting    48.5G   299G      2      0   267K  8.56K
frosting    48.5G   299G    401      0  50.2M      0
frosting    48.6G   299G    384     94  47.9M  7.79M
frosting    48.6G   299G    471      0  58.9M      0
frosting    48.6G   299G    492      0  61.4M      0
frosting    48.6G   299G    393      0  49.0M      0
frosting    48.6G   299G    426      0  53.3M      0
frosting    48.6G   299G    421    147  52.5M  9.71M
frosting    48.6G   299G    507      0  63.4M      0
frosting    48.6G   299G    376      0  47.0M      0
frosting    48.6G   299G    447      0  55.8M      0
frosting    48.6G   299G    433     13  54.2M  1.62M
frosting    48.6G   299G    431     85  53.8M  6.95M
frosting    48.6G   299G    288      0  36.1M      0
frosting    48.6G   299G    329      0  41.2M      0
frosting    48.6G   299G    340      0  42.4M      0
frosting    48.6G   299G    398      9  49.8M  1.14M
frosting    48.6G   299G    324    126  40.4M  7.08M
frosting    48.6G   299G    391      0  48.9M      0
frosting    48.6G   299G    261      0  32.5M      0
frosting    48.6G   299G    314      0  39.3M      0
frosting    48.6G   299G    317      0  39.6M      0
frosting    48.6G   299G    346     79  43.3M  6.36M

Are these "holes" in write speed normal. Since this is the exact symptom we
are getting when the network writes start to be slow.

If I instead copy a large single file, I get this IO behavior:

               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
frosting    50.1G   298G      7      0   953K  34.5K
frosting    50.1G   298G    224    215  27.9M  26.8M
frosting    50.2G   298G    224    364  27.8M  38.6M
frosting    50.2G   298G    225     57  27.9M  7.23M
frosting    50.3G   298G    173    477  21.5M  56.1M
frosting    50.3G   298G    219      0  27.3M      0
frosting    50.3G   298G    265    353  33.0M  44.0M
frosting    50.3G   298G    294    172  36.6M  18.3M
frosting    50.3G   298G    237    436  29.4M  54.2M
frosting    50.4G   298G    257    108  31.9M  9.69M
frosting    50.4G   298G    211    382  26.1M  47.5M
frosting    50.4G   298G    305    162  38.0M  16.4M
frosting    50.4G   298G    253    369  31.5M  45.9M
frosting    50.5G   297G    176    177  21.8M  18.0M
frosting    50.5G   297G    197    167  24.6M  20.9M
frosting    50.6G   297G    248    375  30.9M  42.8M
frosting    50.6G   297G    322    605  39.9M  68.0M
frosting    50.6G   297G    164     36  20.4M  1.57M
frosting    50.6G   297G    259     96  32.2M  12.0M

which looks more like what I would expect and is also similiar to the IO
behavior we get if I copy the folder with many files on an ext4 file system.

Any help or tips for getting this to work would be highly appreciated!

Cheers,
Albert Cervin