Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2)

[Matt Churchyard]

-----Original Message-----
From: Andriy Gapon <avg at FreeBSD.org> 
Sent: 25 March 2021 15:35
To: Matt Churchyard <matt.churchyard at userve.net>; Mathieu Chouquet-Stringer <me+freebsd at mathieu.digital>
Cc: freebsd-fs at freebsd.org
Subject: Re: Scrub incredibly slow with 13.0-RC3 (as well as RC1 & 2)

On 25/03/2021 10:55, Matt Churchyard wrote:
> Just an a aside, I did post a message a few weeks ago with a similar 
> problem on 13 (as well as snapshot issues). Scrub seemed ok for a 
> short while, but then ground to a halt. It would take 10+ minutes to 
> go 0.01%, with everything appearing fairly idle. I finally gave up and stopped it after about 20 hours.
> Moving to 12.2 and
> rebuilding the pool,

>This could make a huge difference, though.
>Scrub performance significantly depends on the data layout.
>The more sequential it is the better.
>So, older (and more fragmented) pools are at disadvantage.

It could, but then this was a brand new install for a backup server and had about 10 datasets sent to it via send/recv. Absolutely no other filesystem usage. It was really a best case for scrub at this point.
When I say it was ok for a while, I don't mean a full scrub completed fine for days/weeks, then started slowing down. I mean the first scrub, day 1, of untouched new data seemed to run ok until around 16%, then just stopped dead for about 6 hours, moving 0.01% every now and then.

The only real difference is that I was hoping to use dataset encryption, which was enabled, but then this shouldn't really affect scrub as the whole point of the openzfs implementation is that encryption doesn't affect anything scrub needs to do its job.

The 12.2 install was done in exactly the same way. A new pool with the same datasets sent to it. As in my previous message, it still scrubs in just over 3 hours even though it has now been receiving snapshots of about 40 datasets, containing about 4 times more data, daily for over a month. Whatever was happening on 13 clearly wasn't hardware related.

Matt

> the system scrubbed the same data in
> an hour, and I've just scrubbed the same system after a month of use 
> with about 4 times the data in 3 hours 20. As far as I'm aware, both 
> should be using effectively the same "new" scrub code.
> 
> Will be interesting if you find a cause as I didn't get any response 
> to what for me was a complete showstopper for moving to 13.


--
Andriy Gapon

--
Andriy Gapon