Jail - PF - NAT - Network Performance

[James Lodge]

> On 25 Oct 2015, at 18:47, James Lodge <James at Lodge.me.uk> wrote:
>
>> On 25 Oct 2015, at 17:46, James Lodge <James at Lodge.me.uk> wrote:
>> I currently have a FreeBSD 10.1 host running on Digital Ocean. I have multiple jails and I'm not using vimage.
>>
>>
>> I'm using PF on the host to NAT traffic from said jails and all is working as expected. I have a jail running OpenVPN and clients can connect and traffic is routed to the Internet down the tunnel via PF/NAT. The issue I'm seeing is download speeds to the client from the Internet on the external side on PF. Upload always seem reasonable, but download is always woeful. I'm using a Windows machine as the client if that make any odds.
>>
>> Yeah, there’s an issue with checksums and pf/Xen.
>> Disabling TSO should work for you (sudo sysctl net.inet.tcp.tso=0), and the problem should be completely fixed in the >next release (10.3 or 11.0)
>>
>> Regards,
>> Kristof
>
>
> Thanks Kristof for the quick reply,
>
> I was hoping it would be that simple, but Digital Ocean use KVM (from what I know) as their hypervisor so disabling TSO and LRO seems to have no noticed increase in performance.
>
>Can you run a tcpdump while you’re seeing the problem?
>The issue I know about is related to the TCP checksum, so I’d expect to see many invalid checksums.
>
>Regards,
>Kristof


There are discussions/threads regarding KVM/TSO with the same symptoms, but I would expect to see the same poor performance through PF be it local or remotely via OpenVPN, but I don't. Local I see perfect performance in and out from the host and jails.  

Running wireshark locally on the windows client and downloading from the Internet via PF, I see a lot of retransmissions and duplicate ACKs. 

28	0.824146000	80.249.99.148	10.8.0.10	TCP	1422	[TCP Retransmission] 80→57292 [ACK] Seq=9577 Ack=1 Win=14 Len=1368

33	0.872164000	10.8.0.10	80.249.99.148	TCP	66	[TCP Dup ACK 29#1] 57292→80 [ACK] Seq=1 Ack=13681 Win=256 Len=0 SLE=15049 SRE=16417


I don't see any retransmissions or duplicate ACKs when downloading from the web server jail via its private IP down the tunnel. 


if I run tcpdump i tun0 -n at the same time as seen the retransmissions and duplicate ACKS I get.....

20:26:58.567017 IP 10.8.0.10.60339 > 80.249.99.148.80: Flags [.], ack 437760, win 256, options [nop,nop,sack 1 {440496:443232}], length 0
20:26:58.568680 IP 80.249.99.148.80 > 10.8.0.10.60339: Flags [.], seq 437760:439128, ack 1, win 14, length 1368
20:26:58.580530 IP 10.8.0.10.60339 > 80.249.99.148.80: Flags [.], ack 439128, win 256, options [nop,nop,sack 1 {440496:443232}], length 0
20:26:58.585141 IP 80.249.99.148.80 > 10.8.0.10.60339: Flags [.], seq 443232:444600, ack 1, win 14, length 1368
20:26:58.596466 IP 10.8.0.10.60339 > 80.249.99.148.80: Flags [.], ack 439128, win 256, options [nop,nop,sack 1 {440496:444600}], length 0

Regards
James