Suggestion for hardware for ZFS fileserver

[Peter Eriksson]

I can give you the specs for the servers we use here for our FreeBSD-based fileservers - which have been working really well for us serving Home directors

Servers:
Dell PowerEdge R730xd
Dell PowerEdge R740xd
HP ProLiant DL380 G9

CPUs
Intel Xeon E5-2620 v4 @ 2.10GHz
Intel Xeon E5-2623 v4 @ 2.60GHz
Intel Xeon E5-2624 v4 @ 2.60GHz
l:
Intel Xeon Silver 4110 CPU @ 2.10GHz
Intel Xeon Silver 4116 CPU @ 2.10GHz

RAM:
256GB RAM

HBA:
LSI SAS3008 (Dell HBA330 Mini, Dell Storage Cntlr.12GB-SASHBA)
HP Smart HBA H241 (configured in “dumb” mode)

Network:
Intel X710 2 & 4-port 10Gb Ethernet adapters 
 (Dell branded ones & HPE Ethernet 10Gb 2-port 562SFP+ Adapter)
We had some trouble getting this to behave well in the beginning so we had to disable TSO via the “-tso” option to ifconfig causing us to lose a bit performance but since we’re nowhere near saturating the 10Gb ethernet links anyway… If HP/Dell had some other 10G ethernet controllers we might have considered switching, but it works “well enough” for now. Hopefully eventually Intel will find and fix the last TSO-related firmware bugs sometime...

Disks:
10GB HGST HUH721010AL4200 (Dell & HP OEM 7200 rpm disks)
 (Requires a low-level reformat to get to 4096-byte sectors as they, when delivered from Dell, have a larger sector size not compatible with the SAS3008 controller with FreeBSD)
+ some SSD:s for SLOG and L2ARC.

OS:
FreeBSD 11.2-RELEASE-p6
  (We discussed using FreeNAS at the beginning but went the “pure” route since we wrote quite a bit of our own management software anyway)

Filesystems setup:
ZFS with RAID-Z2 (14 10TB in 4+2 RAID-Z2 config with 2 spares), SSDs for LOG and L2ARC (Bothe SATA SSDs and M.2 PCIe ones).
ZFS with Mirror for boot disks.

Sharing software:
SMB (SMB v2 or higher only)
  Samba 4.7.6 (we test 4.9.3 right now - but we’re seeing problems with anything later that 4.7.6 currently).

NFS (NFSv4 only, Kerberos/GSS authentication)
  More or less the only thing we’ve tuned for NFS so far is:
     nfsuserd_flags="-manage-gids -domain OURDOMAIN -usertimeout 10 -usermax 100000 16”
  As more clients start using NFS I assume we will have to adjust other stuff too.. Suggestions are welcome :-)

SFTP

We are servicing around 500-600 concurrent SMB users per server, and 1700 users on one (but lighter load), just a few NFS clients right now but more are connecting every day, some SFTP. System joined to our Windows AD (Windows Server 2016) system.

SAS Expanders: 
We only use the built-in disk slots on the Dell servers. We use the HP D6020 external SAS disk shelfs (70 drives per shelf) on the HP servers.


We talked about building our own servers but decided to go the “vendor route” for warranties and stuff. Costs a bit more money but….

- Peter



+ AdapterOn 13 Dec 2018, at 22:59, Sami Halabi <sodynet1 at gmail.com> wrote:
> 
> 
> Hi,
> I'm planning on building a machine for fileserver using ZFS (still
> considering vanilla FreeBSD VS FreeNAS).
> 
> I would like to get recommendation for good and working hardware.
> I have 2 setups I'm interested in getting your help for each.
> 1. Heavy load machine with 10G connection.
> 2. Light machine with 1G connection.
> 
> Both setups will have up to 22 SAS hard drives 2T each + 2 SSD for L2ARC.
> For both setups I would appreciate your help in:
> 1. For raidz2 best strip of vdevs for maximum capacity with reasonable
> performance.
> 2. Hardware recommendations in terms of:
> CPU
> MEMORY
> HBA
> Sas expander for future connecting Jbod
> Server (aka my own build VS brand one like HP/Dell.. Etc).
> 
> Thanks in advance,
> Sami
> _______________________________________________
> freebsd-fs at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe at freebsd.org"