PF frag entries limit reached on a server with hw.ncpu: 24

[Kristof Provost]

On 23 Dec 2019, at 12:00, Andreas Longwitz wrote:
> On one of my servers a saw some messages
>
>    <kern.crit> dssinet kernel:
>    [zone: pf frag entries] PF frag entries limit reached
>
> The output of the command
>    vmstat -z | grep "pf frag entries"
> was
>    pf frag entries:    40,  5000,    0,  5000,  18760,   0,   0
>
> So there are 5000 free items, none is in use and the limit of 5000 is
> reached. This situation is possible, when all the free 5000 items are
> hold in the caches of e.g cpu 0,1..20 and now a thread running on cpu 
> 21
> wants to allocate an item of the zone pf_frent_z. The alloc fails
> because of the limit and the zone allocater from time to time logs the
> error message "limit reached".
>
> After putting the line
>           set limit frags 10000
> in pf.conf the problem was gone, now I have 5700 free items and this
> value did not change for some weeks.
>
> For a server with 24 cpu the default PFFRAG_FRENT_HIWAT = 5000 was too
> small. Maybe this default value should be adjusted.
>
I’m happy to tweak defaults, but I’m not sure there’s a good way 
to figure out what a sane value is.
Possibly we’d want a heuristic based on the RAM size, but that too is 
not ideal: a firewall device will reasonably want to dedicate more of 
its memory to pf than a file server.

I’d prefer to have a holistic view of what defaults make sense for 
these limits before we start changing them.

Best regards,
Kristof