Suspected mbuf leak with Nginx + sendfile + TLS in 12.2-STABLE

[Slawa Olhovchenkov]

On Fri, Feb 05, 2021 at 11:54:07AM +0100, GomoR wrote:

> On 2021-02-05 09:11, GomoR wrote:
> >> The first step I would do if possible would be to bisect between the 
> >> last
> >> known working version and the version that is known to be broken to
> >> determine which commit introduced the problem.  One thing that could 
> >> help
> >> here is to see if you can reproduce the problem using a 12.2 kernel on 
> >> a
> >> 12.1 world + ports.  If you can, then you can limit your bisecting to 
> >> just
> >> building new kernels which will make that process quicker.
> 
> We have reinstalled from scratch our system with FreeBSD 12.1-RELEASE. 
> We then
> have installed just enough of our software stack to reproduce the issue.
> 
> No problem with a stock 12.1-RELEASE kernel, but problem arise after 
> installkernel
> with the latest 12.2-STABLE. We then turned off all our customizations, 
> including
> some specific sysctl.conf values. The bug didn't triggered.
> 
> After dissecting our sysctl values, the faulty one has been identified:
> 
> kern.ipc.maxsockbuf=157286400
> 
> This value is 75 times the default value (2097152). Restoring the 
> default value
> fixes the issue. After some tests, the bug is triggered starting 
> somewhere to
> 64 times the default value (134217728).
> 
> There was no issue with this setting in 12.1-RELEASE, but there is in 
> 12.2-RELEASE.
> 
> Do you have some insights onto why it causes that mbuf problems? In the 
> meantime,
> we have our solution, but we are willing to help identify if that's a 
> kernel bug
> or just a real bad idea to set maxsockbuf to such a high value.

===
> Each time a user downloads a file, mbuf & mbuf_clusters are raising to
> reach the maximum limit in a matter of seconds. Those values are
> asserted by 'netstat -m' as follows:
>
> Normal situation:
>
> mbuf:                   256, 26031105,   16767,    5974,428087938,   0,   0
> mbuf_cluster:          2048, 8135232,   18408,    2704,101644203,   0,   0
>
> Warning situtation:
>
> mbuf:                   256, 26031105, 2981516,  151205,1109483561,   0,   0
> mbuf_cluster:          2048, 8135232, 2983155,    4201,319714617,   0,   0
===

Can you clarified what is problem?
I.e. under load system used more resources and this is not bug.
Do you see more resources usage compared to load?
Or resources don't freed after drop load?