TRIM, iSCSI and %busy waves

Warner Losh imp at bsdimp.com
Fri Apr 6 15:08:09 UTC 2018


On Fri, Apr 6, 2018 at 2:56 AM, Borja Marcos <borjam at sarenet.es> wrote:

>
>
> On 6 Apr 2018, at 10:41, Steven Hartland <killing at multiplay.co.uk> wrote:
>
> That is very hw and use case dependent.
>
> The reason we originally sponsored the project to add TRIM to ZFS was that
> in our case without TRIM the performance got so bad that we had to secure
> erase disks every couple of weeks as they simply became so slow they where
> unusable.
>
> Now admittedly that was a fair few years ago and hw has moved on since
> then, but the point remains that it’s not totally true that just not
> TRIMing is an option.
>
>
> As far as I know, letting the device know that you have released a block
> should be a Good Thing.
> So I prefer to TRIM. I have also seen cases where not doing TRIMs resulted
> in a terrible performance
> although modern SSDs should have better algorithms to deal with it.
>
> But I have also seen bad cases of TRIM requests piling up and clogging the
> queues for long periods
> of time. Admittedly it was a purely synthetic situation (running bonnie++)
> but, again, performing
> operations like destroying a huge snapshot or dataset could trigger a
> similar condition.  It was
> especially nasty when I tried NVMEs. I mentioned this here:
>
> https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023134.html
>
>
> That’s why I think there is a mid point between the radical approach of
> not doing TRIMs at all
> and clogging the queues with too many of them.
>

nvd is especially troublesome. It doesn't do any trim shaping *AT*ALL* and
we machine gun a huge number of TRIM requests when we delete large files.
TRIM is normally not a fast path operation inside the drives, so sending
lots of TRIMs down will often starve read/write resources for a variety of
reasons (it often forces single threading limiting the parallelism in the
drive, it often kicks off GC which causes lots of block erases which are
slow and block reads/writes to other pages/blocks on the die (some or all
mitigated by planes, but still), etc.


> If the TRIM queue gets very large and the “mandatory” operations (read,
> writes, etc) begin to
> get large delays I think it’s safe to assume that *there is* a problem and
> discarding at least part
> of those TRIMs could help to solve it.
>

That's often a property of the devices, however. And how quickly we shove
TRIMs down its throat. I've seen situations where we play out the TRIMs
slowly enough that we can see huge latencies in TRIMs (seconds to minutes),
but see little to no effect effect on read latency, even in the P99 number
since that's often where effects in TRIMs show up and are easiest to
measure.

Warner


> A “TRIM when you can” should still be better than a “Don’t TRIM at all”.
>
> Cheers,
>
>
>
>
>
> Borja.
>
> P.S: Attaching the graphs that were lost.
>
>
>
>
>
>


More information about the freebsd-stable mailing list