From nobody Sat Oct 23 18:42:35 2021 X-Original-To: freebsd-pf@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 60CB7180BFCE for ; Sat, 23 Oct 2021 18:42:44 +0000 (UTC) (envelope-from marcel@herrbischoff.com) Received: from mailpod.herrbischoff.com (mailpod.herrbischoff.com [157.90.240.191]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mailpod.herrbischoff.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Hc97l1chWz52JG for ; Sat, 23 Oct 2021 18:42:39 +0000 (UTC) (envelope-from marcel@herrbischoff.com) Received: from mailpod.herrbischoff.com (localhost [127.0.0.1]) by mailpod.herrbischoff.com (OpenSMTPD) with ESMTP id 704854de for ; Sat, 23 Oct 2021 20:42:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed; d=herrbischoff.com; h=date :from:to:subject:message-id:mime-version:content-type; s=hrbf; bh=ed8vZrWWLMIk2r4bURO9VJt4pIUraHIi6jAdTEFDGnM=; b=dzBBAtzns9Zx nc13Ce8DYQOpf2co5kBqnyOW84eXJPPGeUrmJqMEaYjYv11ItfCWI2hkqnThgs1h AKZqZoYwm9ThBlzIRzUdr5sjjtYpKvj5198aIuD+v2axPZtlNI/pSD0y26GY7ymn V5gquoZ7ECpBc37JAlGobeF7J1y7+moKNQTquTAHChI0Yoptb6iIOITwwH+/zrA8 XaD1IYDnY9JHRJkxIPChTk1xZIWrtdbekUahBIVgAgGOTIv8DSyH693sKFS6qO8c FNnwmEh722dw/7iXNOtRM6lvuGzZgrnHLX3qH/wmTDCmIzs/l+BVskUfOFmucVsS NfNSbrRQDg== DomainKey-Signature: a=rsa-sha1; c=nofws; d=herrbischoff.com; h=date :from:to:subject:message-id:mime-version:content-type; q=dns; s= hrbf; b=nwkRJxTtcEihjRL9oPwgSu6mPMj/qQ4IguewyxvfxqpxQYDjbJzGldMM u5195Lg2Y5KEG8wIO9FjhA2Z49qaGc3eO4nHO7YXfWa9JVImH29/Y+ehP6/HvK/M ZAAi+mEtNIkQWzZ6wZAfKEiukGLObanzjCCSeuS6DcIbikPSFhycYgyc5Rzbl3LQ qgPGVgX0mg6bCmJxvXUIQcHuqvX9GX6EAb50PJ5rHzSgy1SPX+dHoCzjc1p6GjuQ WMFE15QK/euZfky8wRZMnFP3kTe1mh8BposxsQMZabBL1J7yVa7JJysE3xKPr2TX 5+BwXHNET8jCja59PHQCuRs65pnvlQ== Received: by mailpod.herrbischoff.com (OpenSMTPD) with ESMTPSA id 763810ef (TLSv1.3:AEAD-AES256-GCM-SHA384:256:NO) auth=yes user=marcel@herrbischoff.com for ; Sat, 23 Oct 2021 20:42:36 +0200 (CEST) Date: Sat, 23 Oct 2021 20:42:35 +0200 From: Marcel Bischoff To: freebsd-pf@FreeBSD.org Subject: "pfctl: Cannot allocate memory" issue with a large table Message-ID: List-Id: Technical discussion and general questions about packet filter (pf) List-Archive: https://lists.freebsd.org/archives/freebsd-pf List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-pf@freebsd.org X-BeenThere: freebsd-pf@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline X-Rspamd-Queue-Id: 4Hc97l1chWz52JG X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=herrbischoff.com header.s=hrbf header.b=dzBBAtzn; dmarc=pass (policy=none) header.from=herrbischoff.com; spf=pass (mx1.freebsd.org: domain of marcel@herrbischoff.com designates 157.90.240.191 as permitted sender) smtp.mailfrom=marcel@herrbischoff.com X-Spamd-Result: default: False [-6.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[herrbischoff.com:s=hrbf]; FREEFALL_USER(0.00)[marcel]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_MED(-2.00)[herrbischoff.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-pf@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_THREE(0.00)[3]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DKIM_TRACE(0.00)[herrbischoff.com:+]; DMARC_POLICY_ALLOW(-0.50)[herrbischoff.com,none]; NEURAL_HAM_SHORT(-1.00)[-0.999]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:157.90.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Hi all, for some time now I'm using the excellent (in my opinion) pf-badhost script [https://geoghegan.ca/pfbadhost.html] to create default blocklists for some servers. When using IPv6 and/or geoblocking with it, I often run into the "pfctl: Cannot allocate memory" error upon replacing the table contents. The list contains about 300k+ lines with IPs and CIDRs. It is properly aggregated so the net blocks are compacted as far as possible into CIDR notation. Only single IPs are listed without a "/32" or "/128" suffix. /etc/pf.conf contains > set limit table-entries 1000000 /boot/loader.conf contains > net.pf.request_maxcount=1000000 > kern.maxdsiz="2147483648" /etc/sysctl.conf contains > net.pf.request_maxcount=1000000 "pfctl -s memory" shows the limit is active: > states hard limit 100000 > src-nodes hard limit 10000 > frags hard limit 5000 > table-entries hard limit 1000000 During my research I found out that replacing a pf table temporarily needs double the memory as both the old and new states are held before the old is discarded. This makes entirely sense to me. What I don't understand is why the error still occurs despite the proper limit being set. Does anyone have an idea how I can resolve this? It is entirely possible this happens due to me not entirely understanding how memory allocation in pf works. However, I haven't found anything particularly applicable either in the Handbook or the "pf.conf" man page. Best, Marcel