performance issue within VNET jail

[Michael Grimm]

Hi

[ I did recently migrate my servers from bare metal to cloud instances (OpenStack at OVH) ]
[ FreeBSD 11.1-STABLE #0 r327055                                                          ]


My setup is as follows and didn't change for the last couple of years:

	extIF0/pf/NAT <—> epairXa (bridge0) epairXb <-> jail

Downloading a file (by wget) at the host is around 30 MB/s, and an example tcpdump at extIF0 looks as follows:

19:32:10.711769 IP (tos 0x20, ttl 56, id 37539, offset 0, flags [DF], proto TCP (6), length 8680)
    remote.http > myhost.14367: Flags [.], cksum 0x64ed (incorrect -> 0x3223), seq 5753:14381, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 8628: HTTP
19:32:10.713851 IP (tos 0x20, ttl 56, id 37545, offset 0, flags [DF], proto TCP (6), length 1490)
    remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> 0x8d1e), seq 14381:15819, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 1438: HTTP
19:32:10.713899 IP (tos 0x20, ttl 56, id 37546, offset 0, flags [DF], proto TCP (6), length 1490)
    remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> 0x6ade), seq 15819:17257, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 1438: HTTP
19:32:10.713934 IP (tos 0x20, ttl 56, id 37547, offset 0, flags [DF], proto TCP (6), length 1490)
    remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> 0x1173), seq 17257:18695, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 1438: HTTP
19:32:10.713962 IP (tos 0x20, ttl 56, id 37548, offset 0, flags [DF], proto TCP (6), length 1490)
    remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> 0xcf7a), seq 18695:20133, ack 146, win 235, options [nop,nop,TS val 1007145732 ecr 3995852], length 1438: HTTP


When downloading the very same file within a VIMAGE jail the performance drops to around 80 KB/s, quite a dramatic loss. An example tcpdump at exitIF0 looks as follows:

19:34:36.284175 IP (tos 0x0, ttl 56, id 28618, offset 0, flags [DF], proto TCP (6), length 2948)
    remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> 0x4478), seq 1449:4345, ack 146, win 235, options [nop,nop,TS val 1007182125 ecr 4141429], length 2896: HTTP
19:34:36.481904 IP (tos 0x0, ttl 56, id 28620, offset 0, flags [DF], proto TCP (6), length 1500)
    remote.http > myhost.63382: Flags [.], cksum 0xd11d (correct), seq 1449:2897, ack 146, win 235, options [nop,nop,TS val 1007182175 ecr 4141429], length 1448: HTTP
19:34:36.484109 IP (tos 0x0, ttl 56, id 28621, offset 0, flags [DF], proto TCP (6), length 2948)
    remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> 0x2e5b), seq 15929:18825, ack 146, win 235, options [nop,nop,TS val 1007182175 ecr 4141629], length 2896: HTTP
19:34:36.682006 IP (tos 0x0, ttl 56, id 28623, offset 0, flags [DF], proto TCP (6), length 1500)
    remote.http > myhost.63382: Flags [.], cksum 0x4ab6 (correct), seq 2897:4345, ack 146, win 235, options [nop,nop,TS val 1007182225 ecr 4141629], length 1448: HTTP
19:34:36.684159 IP (tos 0x0, ttl 56, id 28624, offset 0, flags [DF], proto TCP (6), length 2948)
    remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> 0xd7db), seq 18825:21721, ack 146, win 235, options [nop,nop,TS val 1007182225 ecr 4141829], length 2896: HTTP

A tcpdump at epairXa looks comparable.

I did reduce all MTU settings at the involved interfaces from their initial settings (1490) to an experimental setting of 1400, just to be on the save side, to no avail. (FYI: I did have to reduce from 1500 to 1490 to please IPSec after migration from bare metal to cloud infrastructure.)

Then, I did test the following settings found in the Net, to no avail either:

	sysctl net.inet.tcp.tso=0
	sysctl net.link.bridge.pfil_onlyip=0
	sysctl net.link.bridge.pfil_bridge=0
	sysctl net.link.bridge.pfil_member=0
	sysctl net.add_addr_allfibs=0


I do have to admit that I am lost here, and that I cannot think about what is going wrong. The last download I did try at my old severs has been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and I did move my infrastructure from bare metal to cloud, thus I cannot test anymore if my old servers would have shown that performance issue in the meantime.

Thus any feedback is highly recommended!

Thanks in advance and regards,
Michael