Commit r345200 (new ARC reclamation threads) looks suspicious to me - second potential problem

[Slawa Olhovchenkov]

On Wed, May 22, 2019 at 12:49:48PM -0400, Alexander Motin wrote:

> On 22.05.2019 12:19, Slawa Olhovchenkov wrote:
> > On Wed, May 22, 2019 at 12:07:29PM -0400, Alexander Motin wrote:
> > 
> >> On 22.05.2019 11:50, Lev Serebryakov wrote:
> >>> On 22.05.2019 18:19, Alexander Motin wrote:
> >>>
> >>>>>>  But looks like `arc_kmem_reap_soon()` is synchronous on FreeBSD! So,
> >>>>>> this `delay()` looks very wrong. Am I right?
> >>>>
> >>>> Why is it wrong?
> >>>  One second pause after synchronous operation to wait it completion?
> >>
> >> No.  To rate-throttle them.  This gives UMA a second to get back into
> >> minimally steady state after we ripped all caches from it.  As I have
> >> told, we do not want to drain caches constantly in a tight loop, we want
> >> more or less steady state.
> > 
> > And also (posible) additionaly delay arc_get_data_impl().
> 
> arc_get_data_impl() depends on arc_adjust_zthr, not on arc_reap_zthr, so
> it should not get blocked by this delay.  That was the motivation for
> the threads splitting in the last rewrite.

next case: system under memory pressure, no memory in UMA cache.

arc_get_data_impl() see arc_size >= arc_c+overflow (arc_is_overflowing()) and
wait arc_adjust_zthr.

arc_adjust_zthr in arc_adjust_cb() do arc_adjust() and evict small
amount, arc_size still over arc_c (I mean this is posible: arc_c drop by arc_lowmem or arc_szie rised by paralel arc_get_data_impl()),
cv_broadcast() don't called.

next round is wait work arc_reap_zthr activated by arc_lowmem or
timeout:

arc_kmem_reap_soon();
dealy();
arc_reduce_target_size() (under memory pressure (arc_c >> arc_shrink_shift) > arc_available_memory() is true)

arc_reduce_target_size() re-activate arc_adjust_zthr (asize > arc_c is true).

delay() in this path unnecessary and slowly arc_get_data_impl().

> > This is incorrectly throttling implementation.
> 
> I am not particularly defending ZFS doing its own reclamation, I'd more
> trust pagedaemon, but so far I haven't seen any memory pressure issues
> after I committed this.
> 
> -- 
> Alexander Motin