[FreeBSD 8.2 amd64 XENHVM] Incorrect behavior of xen networking
drivers: unexpected tcp reassembling
ivan.ivanyuk at gmail.com
Wed Jan 11 23:04:26 UTC 2012
I've been running FreeBSD 8.2-RELEASE amd64 with XENHVM kernel with PV
drivers for a couple of months but when trying to use it as router I
get huge problems with network performance.
I can get about 500Mb/s from virtual machine to external host, but
only 200Kb/s from any internal PC to the same external host through
virtual machine router.
Closest description I found in archives is this:
With further investigation by tcpdump I found a couple of problems
with packets handling.
My setup is like this:
Dom0 is Debian with xen-hypervisor-4.0-amd64 4.0.1-2.
DomU is FreeBSD 8.2-RELEASE amd64 XENHVM kernel with "device xenpci" option
--------- --------------- -------------- ---------------
|Internet|<--->|eth3 vif14.0|<--->|xn0 xn1|<--->|vif14.1
---------- | inetbr0 | | Freebsd | | localbr0 |
| Dom0 | | DomU | | Dom0 |
--------------- -------------- ---------------
So I can see couple of packets with TCP data from Internets coming to
eth3, then the same packets are seen on vif14.0. And then on xn0 I see
only one packet with reassembled TCP payload.
While these big (2976 bytes, 4464 bytes, 8928 bytes, etc) packets are
addressed to DomU - all works. When we try to route them elsewhere -
we get ICMP need fragmentation message sent to origin of these
packets. That's because original (small) TCP packets have DF flag set.
And resulting big TCP packet has DF flag as well.
So it seems to me that something in the chain "vif14.0<->xn0" is
reassembling TCP packets.
Is there some sysctl or other settings to control this behavior? (I
tried turning off all offloads on vif14.0 in Dom0, tried to change
fragmentation settings in FreeBSD, nothing changed)
Now I set "scrub no-df" in pf to permit fragmentation of this big
packets but get another strange problem, which still prevent expected
network performance, but this will wait until first problem is solved.
I can provide tcpdumps from eth3, vif14.0, xn0 for investigation.
More information about the freebsd-xen