dead slow update servers
hw at adminart.net
Tue Jul 16 00:22:59 UTC 2019
"Kevin P. Neal" <kpn at neutralgood.org> writes:
> On Mon, Jul 15, 2019 at 06:09:25PM +0200, hw wrote:
>> "Kevin P. Neal" <kpn at neutralgood.org> writes:
>> > On Mon, Jul 15, 2019 at 05:42:25AM +0200, hw wrote:
>> >> "Kevin P. Neal" <kpn at neutralgood.org> writes:
>> >> > Oh, and my Dell machines are old enough that I'm stuck with the hardware
>> >> > RAID controller. I use ZFS and have raid0 arrays configured with single
>> >> > drives in each. I _hate_ it. When a drive fails the machine reboots and
>> >> > the controller hangs the boot until I drive out there and dump the card's
>> >> > cache. It's just awful.
>> >> That doesn't sound like a good setup. Usually, nothing reboots when a
>> >> drive fails.
>> >> Would it be a disadvantage to put all drives into a single RAID10 (or
>> >> each half of them into one) and put ZFS on it (or them) if you want to
>> >> keep ZFS?
>> > Well, it still leaves me with the overhead of dealing with creating arrays
>> > in the hardware.
>> Didn't you need to create the RAID0s having a single disk, too?
> Yes, and that wouldn't change if I took your suggestion. Which is why I
> phrased it as "it still leaves" which says that the problem already exists
> and would continue to exist.
So it's not more work as otherwise --- maybe even less because you have
only half the number of logical drives, or less.
>> > And it costs me loss of the scrubbing/verification of the end-to-end
>> > checksumming. So I'm less safe there with no less work.
>> If you're worried about the controller giving results that lead to the
>> correct check sums and data ending up on the disk not matching these
>> check sums when the controller reads it later, what difference does it
>> make which kind of RAID you use? You can always run a scrub to verify
>> the check sums, and if errors are being found, you may need to replace
>> the controller.
> ZFS can correct checksum errors so long as the array is still valid. Is
> there a hardware RAID card that does that?
I'm not sure, some controllers do what they call surface checking in the
background. It's the job of the controller to make sure the data on the
disk is fine, and I have no reason to assume that it doesn't do that.
>> > Maybe I should just go ahead and change it. I've got a drive about to
>> > fail on me. It's a three way mirror so I'm not worried about it. It would
>> > be, uh, _nice_ if it didn't bring down the machine, though.
>> If you were using two or more disks each in a RAID1 or RAID10 to create
>> one disk exposed to ZFS, you wouldn't have a problem when one disk
>> becomes unresponsive. If there's someone around who is used to quickly
> True, but if I run a scrub I want to verify _all_ the data on the disks,
> not just the data exposed by the RAID controller card.
Why? Data that is never exposed is irrelevant for this and doesn't need
to be verified.
Of the logical volumes you're using now, the controller card exposes no
more than it does, so what difference does it make of how many disks the
volume is made of? You don't get any more or less data exposed as the
controller does its thing regardless.
Besides, how do you suggest to get all data exposed? Do you have
special firmware on your disks that exposes all data, and for everything
else that is involved?
> That means ZFS needs to see _all_ the disks. Otherwise I would be
> vulnerable to loss of data due to checksum errors that may only be
> seen after a drive dies. At that point it's too late to correct.
I don't understand how it would make a difference whether ZFS can see
all physical disks or not. One way or another, there is merely a
storage device ZFS is using and can see, and if the device is
errorneous, it shouldn't matter what the device is physically made of.
When a disk fails and is replaced, the hardware RAID is being rebuilt,
and when ZFS detects checksum errors on that storage device, why
shouldn't it be able to correct the errors like with any other storage
device? It's even not the storage device that failed but only a disk.
What if you had hybrid disks consisting of some flash memory and
magnetic disks? You would have to conclude that they can never be used
with ZFS because they could introduce hidden checksum errors. What
about the cache of your disks, do you turn it off because hidden
checksum errors could be introduced when there is a mismatch between the
data on the disk and what is being exposed by the cache? IIRC, one of
the advantages of ZFS is that you don't need to disable the cache.
RAID controllers and such aren't designed to create hidden checksum
errors ZFS is unable to correct. ZFS was designed to detect errors and
to correct them, using check sums. There's no need to be afraid of RAID
controllers and such.
>> Hardware RAID does have advantages, so why not use them when you're
>> stuck with it anyway?
> Because the downsides outweigh the upside.
Like how? The whole setup seems very questionable, and the machine goes
out of service right away when a disk becomes unresponsive. One of the
purposes of RAID is to prevent just that.
More information about the freebsd-questions