Suggestion for hardware for ZFS fileserver

[Karl Denninger]

On 12/24/2018 17:13, Zaphod Beeblebrox wrote:
> [ regarding ZFS hardware thread ]
>
> There's another type of server --- the "ghetto" or home storage serer.  For
> this server, I like to optimize for not loosing data, not for uptime.
>
> Going back a few years, there were consumer motherboards with 10 or 12 SATA
> onboard.  Mostly, this was at the change of technologies ... so you had
> some of one kind of port and some of another.  Used SAS HBAs are another
> option ... but they have a caviat: many SATA drives will eventually reject
> them under load.  Good SATA drives won't (but again, we're talking a ghetto
> system).  If you're taking WD reds (and not, say, seagate barracudas) ...
> these work well.  On the seagates, however, I've had drives repeatedly fail
> ... only to go on working fine in a workstation with a SATA controller.

I've run "ghetto mode" fileservers with the LSI adapters in IT mode
(that always just seem to work) with one of their SFP ports connected to
a SAS expander, and then fanned THAT out to SATA drives.  The only
constraint is that you can run into problems booting from an expander,
so don't -- use the ports on the HBA (or even the motherboard) for the
boot drives.

Never had a problem doing this with HGST drives, Intel SSDs and most
others.  The Seagates I've had fail actually physically failed; they
didn't throw a protocol hissy fit on the bus.  I don't buy Seagates any
more as I've had too many die out-of-warranty for my taste.  They work
fine with WD drives too.  Never had one of the drives that failed cause
a cascade detach event either.  The last few years (five or so) for
spinning rust HGST seems to sell the most-reliable stuff in my
experience but YMMV on that.

Those adapters and expanders are cheap these days.  The expanders used
to be expensive, but not any more -- there's a ton of them around on the
secondary market for very little money (not much more than the LSI
cards.)  Their only downside is they run hot so you need good fan
coverage in the case.

Older SuperMicro boards (X8DTL- series) that will take the 5600-series
Westmere Xeon processors can be had for almost nothing (note you have to
have the latest BIOS in them, which can be flashed, to run the Westmere
processors), and the CPUs are a literal $25.  The only "gotcha" is you
need ECC memory, but if you can find it at a decent price used you're
golden.  I would NOT run a ZFS filesystem without ECC memory; the risk
of undetected data corruption that you don't catch for months or even
years is material and if it happens you WILL cry since your backup
coverage may have expired at that point.

> Last point.  RAID-Z2 at a minimum.  I could even see the argument for Z3.
> My current array is 16x 4T drvies in to 8 disk Z2 plexes.  Of that, one
> plex is all WD Red on a SAS controller ... and the other (older) plex is
> still largely cheap drives on SATA.  Right now, drives below 4T are
> artificially expensive.  Drives right up to 10T are about the same price
> per G (at least here in Canada).

Yes, RaidZ2 is a (very) good idea.

Oh, and keep a spare power supply in the building, plus a spare disk
adapter and expander.  Hell, for $25 you can keep a spare CPU in the
building! :)  A box like this can trivially saturate more than one GigE
port (the SuperMicros typically have two on board plus the dedicated
management port) all day long.  While there are failure points in such a
ghetto system config they're coverable (with a reboot after swapping the
dead item) at very low cost in backstock in the building.

I've had a couple of these in 24x7 field use for coming up on ten years
now -- they're still running and have been dead-balls reliable --
duplicating them today is disgustingly inexpensive.  If there's one
downside its that modern CPUs are FAR more power-efficient per-cycle
than the older Xeons (they're much faster too, but that's not the
limiting factor for a fileserver.)

-- 
Karl Denninger
karl at denninger.net <mailto:karl at denninger.net>
/The Market Ticker/
/[S/MIME encrypted email preferred]/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4897 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.freebsd.org/pipermail/freebsd-fs/attachments/20181224/ae2fff3f/attachment.bin>