Best practice for virtualized pf based NAT router?

[Kristof Provost]

On 4 Oct 2016, at 13:02, Trond Endrestøl wrote:
> On Tue, 4 Oct 2016 12:19+0200, Kristof Provost wrote:
>> ifconfig xn0 -rxcsum -txcsum -rxcsum6 -txcsum6 -tso6 -tso4 -lro (and 
>> the same
>> for xn1).
>
> That made all the difference. Thank you.
>
Bah. I was hoping I’d put that bug to rest.

>> If that makes a difference I’d be very interested in both network 
>> captures and
>> further debugging.
>
> I'm pretty sure you ment if your proposed changes _doesn't_ make any
> difference, but if you want network captures, etc, I'm sure I can
> arrange it.
>
No, I meant if this helped. It means that a bug I thought was fully 
fixed is still there.

The fix was done in r289316:

     pf: Fix TSO issues

     In certain configurations (mostly but not exclusively as a VM on 
Xen) pf
     produced packets with an invalid TCP checksum.

     The problem was that pf could only handle packets with a full 
checksum. The
     FreeBSD IP stack produces TCP packets with a pseudo-header checksum 
(only
     addresses, length and protocol).
     Certain network interfaces expect to see the pseudo-header 
checksum, so they
     end up producing packets with invalid checksums.

     To fix this stop calculating the full checksum and teach pf to only 
update TCP
     checksums if TSO is disabled or the change affects the 
pseudo-header checksum.

     PR:             154428, 193579, 198868
     Relnotes:       yes
     Sponsored by:   RootBSD


It’s great that you’ve got a workaround, but the problem should be 
completely gone, and it’s clearly not.

If you’re willing to spend a bit more time on this I’d like to dig 
into it a bit, and try to find out what I missed.

Let’s start by looking at the network capture (with the offloads 
turned back on, so we can reproduce the problem).
I expect we’ll see incorrect TCP checksums, which is the cause of your 
bad performance.

It’s slightly surprising that it only happens in the forwarding path, 
but at least that’s something to go on.

Regards,
Kristof