Recommendations for servers running SATA drives
spork at bway.net
Sat Sep 27 19:42:53 UTC 2008
I'm forking the thread on fsck/soft-updates in hopes of getting some
practical advice based on the discussion here of background fsck,
softupdates and write-caching on SATA drives.
On Fri, 26 Sep 2008, Jeremy Chadwick wrote:
> Let's be realistic. We're talking about ATA and SATA hard disks, hooked
> up to on-board controllers -- these are the majority of users. Those
> with ATA/SATA RAID controllers (not on-board RAID either; most/all of
> those do not let you disable drive write caching) *might* have a RAID
> BIOS menu item for disabling said feature.
While I would love to deploy every server with SAS, that's not practical
in many cases, especially for light-duty servers that are not being pushed
very hard. I am taking my chances with multiple affordable drives and
gmirror where I cannot throw in a 3Ware card. I imagine that many
non-desktop FreeBSD users are doing the same considering you can fetch a
decent 1U box with plenty of storage for not much more than $1K. I assume
many here are in agreement on this point -- just making it clear that the
bargain crowd is not some weird edge case in the userbase...
> Regardless of all of this, end-users should, in no way shape or form,
> be expected to go to great lengths to disable their disk's write cache.
> They will not, I can assure you. Thus, we must assume: write caching
> on a disk will be enabled, period. If a filesystem is engineered with
> that fact ignored, then the filesystem is either 1) worthless, or 2)
> serves a very niche purpose and should not be the default filesystem.
Arguments about defaults aside, this is my first questions. If I've got a
server with multiple SATA drives mirrored with gmirror, is turning on
write-caching a good idea? What kind of performance impact should I
expect? What is the relationship between caching, soft-updates, and
either NCQ or TCQ?
Here's an example of a Seagate, trimmed for brevity:
Protocol Serial ATA v1.0
device model ST3160811AS
Feature Support Enable Value Vendor
write cache yes yes
read ahead yes yes
Native Command Queuing (NCQ) yes - 31/0x1F
Tagged Command Queuing (TCQ) no no 31/0x1F
TCQ is clearly not supported, NCQ seems to be supported, but I don't know
how to tell if it's actually enabled or not. Write-caching is currently
The tradeoff is apparently performance vs. more reliable recovery should
the machine lose power, smoke itself, etc., but all I've seen is anecdotal
evidence of how bad performance gets.
FWIW, this machine in particular had it's mainboard go up in smoke last
week. One drive was too far gone for gmirror to rebuild it without doing
a "forget" and "insert". The remaining drive was too screwy for
background fsck, but a manual check in single-user left me with no real
suprises or problems.
> The system is already up and the filesystems mounted. If the error in
> question is of such severity that it would impact a user's ability to
> reliably use the filesystem, how do you expect constant screaming on
> the console will help? A user won't know what it means; there is
> already evidence of this happening (re: mysterious ATA DMA errors which
> still cannot be figured out).
> IMHO, a dirty filesystem should not be mounted until it's been fully
> analysed/scanned by fsck. So again, people are putting faith into
> UFS2+SU despite actual evidence proving that it doesn't handle all
I'll ask, but it seems like the consensus here is that background fsck,
while the default, is best left disabled. The cases where it might make
-servers that have incredibly huge filesystems (and even there being able
to selectively background fsck filesystems might be helpful)
The first example is obvious, people want a fast-booting desktop. The
second is trading long fsck times in single-user for some uncertainty.
> The problem here is that when it was created, it was sort of an
> "experiment". Now, when someone installs FreeBSD, UFS2 is the default
> filesystem used, and SU are enabled on every filesystem except the root
> fs. Thus, we have now put ourselves into a situation where said
> feature ***must*** be reliable in all cases.
> You're also forgetting a huge focus of SU -- snapshots. However, there
> are more than enough facts on the table at this point concluding that
> snapshots are causing more problems than previously expected. And
> there's further evidence filesystem snapshots shouldn't even be used in
> this way.
> Filesystems have to be reliable; data integrity is focus #1, and cannot
> be sacrificed. Users and administrators *expect* a filesystem to be
> reliable. No one is going to keep using a filesystem if it has
> disadvantages which can result in data loss or "waste of administrative
> time" (which I believe is what's occurring here).
The softupdates question seems tied quite closely to the write-caching
question. If write-caching "breaks" SU, that makes things tricky. So
another big question:
If write-caching is enabled, should SU be disabled?
And again, what kind of performance and/or reliability sacrifices are
I'd love to hear some input from both admins dealing with this stuff in
production and from any developers who are making decisions about the
future direction of all of this.
> : http://www.usenix.org/publications/library/proceedings/bsdcon02/mckusick/mckusick_html/index.html
> : http://wiki.freebsd.org/JeremyChadwick/ATA_issues_and_troubleshooting
> : http://wiki.freebsd.org/JeremyChadwick/Commonly_reported_issues
> : http://lists.freebsd.org/pipermail/freebsd-stable/2007-January/032070.html
> | Jeremy Chadwick jdc at parodius.com |
> | Parodius Networking http://www.parodius.com/ |
> | UNIX Systems Administrator Mountain View, CA, USA |
> | Making life hard for others since 1977. PGP: 4BD6C0CB |
> freebsd-stable at freebsd.org mailing list
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe at freebsd.org"
More information about the freebsd-stable