Suspected mbuf leak with Nginx + sendfile + TLS in 12.2-STABLE

[John Baldwin]

On 2/4/21 8:08 AM, GomoR wrote:
> Dear FreeBSD community,
> 
> we are encountering a DoS condition on our production machines.
> Our use case is an Nginx reverse proxy serving large files via HTTPS.
> This problem arose when switching kernel and userland from 12.1-RELEASE
> to 12.2-RELEASE. Ports were not upgraded (at first).
> 
> Each time a user downloads a file, mbuf & mbuf_clusters are raising to
> reach the maximum limit in a matter of seconds. Those values are
> asserted by 'netstat -m' as follows:
> 
> Normal situation:
> 
> mbuf:                   256, 26031105,   16767,    5974,428087938,   0,
>    0
> mbuf_cluster:          2048, 8135232,   18408,    2704,101644203,   0,
> 0
> 
> Warning situtation:
> 
> mbuf:                   256, 26031105, 2981516,  151205,1109483561,   0,
>     0
> mbuf_cluster:          2048, 8135232, 2983155,    4201,319714617,   0,
> 0
> 
> We have seen a patch related to sendfile + KTLS + mbuf at the below link
> and we updated to -STABLE to apply:

None of the sendfile or KTLS changes from Netflix are in 12, they are only
in 13 and later.

> Don't transmit mbufs that aren't yet ready on TOE sockets.
> This includes mbufs waiting for data from sendfile() I/O requests, or
> mbufs awaiting encryption for KTLS.
> https://github.com/freebsd/freebsd-src/commit/14c77f30b201bf76119d59678e72051c093333c2

This patch only applies to Chelsio T5/T6 NICs when using TOE (TCP offload)
and doesn't affect freeing mbufs, it just fixes a race when the NIC could
potentially send random garbage if it sends the mbuf before the scheduled
disk I/O to populate it with data from disk has completed.

> NIC is:
> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver>
> 
> What can we do to help you find the root cause?

The first step I would do if possible would be to bisect between the last
known working version and the version that is known to be broken to
determine which commit introduced the problem.  One thing that could help
here is to see if you can reproduce the problem using a 12.2 kernel on a
12.1 world + ports.  If you can, then you can limit your bisecting to just
building new kernels which will make that process quicker.

You might also see if using a different NIC shows the same problem.  If
not, then it might point to a regression in the NIC driver (or perhaps in
iflib as ix uses iflib I believe).

-- 
John Baldwin