Re: "pfctl: Cannot allocate memory" issue with a large table

On 2021-10-23 11:42, Marcel Bischoff wrote:
> Hi all,
> 
> for some time now I'm using the excellent (in my opinion) pf-badhost script
> [https://geoghegan.ca/pfbadhost.html] to create default blocklists for some
> servers. When using IPv6 and/or geoblocking with it, I often run into the 
> "pfctl:
> Cannot allocate memory" error upon replacing the table contents.
> 
> The list contains about 300k+ lines with IPs and CIDRs. It is properly 
> aggregated
> so the net blocks are compacted as far as possible into CIDR notation. Only 
> single
> IPs are listed without a "/32" or "/128" suffix.
> 
> /etc/pf.conf contains
> 
>> set limit table-entries 1000000
> 
> /boot/loader.conf contains
> 
>> net.pf.request_maxcount=1000000
>> kern.maxdsiz="2147483648"
> 
> /etc/sysctl.conf contains
> 
>> net.pf.request_maxcount=1000000
> 
> "pfctl -s memory" shows the limit is active:
> 
>> states        hard limit   100000
>> src-nodes     hard limit    10000
>> frags         hard limit     5000
>> table-entries hard limit  1000000
> 
> During my research I found out that replacing a pf table temporarily needs 
> double
> the memory as both the old and new states are held before the old is 
> discarded.
> This makes entirely sense to me. What I don't understand is why the error 
> still
> occurs despite the proper limit being set.
> 
> Does anyone have an idea how I can resolve this? It is entirely possible 
> this
> happens due to me not entirely understanding how memory allocation in pf 
> works.
> However, I haven't found anything particularly applicable either in the 
> Handbook
> or the "pf.conf" man page.
Have you reached your STATE limit?
OTOH you might try adding the IPs from the list individually. Something like:

iplist="
w.x.y.z
a.b.c.d
...
g.h.i.j
"

for block in $iplist
do
	pfctl -T add -t <your-table-name-here> $block
done

I'm managing about a half dozen tables with a combined number of a over
quarter of a billion addresses, and don't have a problem. Even on a servers
with as little as 8GB RAM.

HTH

--Chris

> 
> Best,
> Marcel