[FreeBSD 8.2 amd64 XENHVM] DomU terrible network performance trought NAT

[Laurent Cligny]

Hello all,

I want to setup a FreeBSD DomU VM as a network NAT gateway in order to
provide Internet access to other FreeBSD and Linux DomU VMs.
My Dom0 is Xen Cloud Platform 1.0 on a Dell Poweredge 210 with 8 cores,
16GiB RAM and one NIC.

All FreeBSD VM are 8.2 amd64 with XENHVM kernel anf the Linux VM is a
Paravirtualized Debian amd64.

The NAT gateway is a two-NIC box, with one (xn0) configured with a
public Internet address and the other (xn1) configured with a class A
private address (10.0.0.254).
The Internet traffic is very good from my FreeBSD NAT gateway to the
Internet (~ 50MiB/s) and also between other VMs on the private network
(~ 70MiB/s trough scp) after applying the patch here
(http://www.mail-archive.com/freebsd-xen@freebsd.org/msg00855.html)
fixing the "too many frags" problem.

Here is a schema of my setup with the observed net speed on it:

----------         ---------------         ----------------------
|Internet|---xn0---|FBSD DomU NAT|---xn1---|FBSD and Linux DomUs|
----------         ---------------         ----------------------
          <------->               <------->
           50MiB/s                 70MiB/s

          <------------------------------->
           0KiB/s <network speed< 5 KiB/s

I tried doing NAT on the FBSD gateway with PF and also with NATd after
shutting down PF and unloaded the kernel modules pf.ko and pflog.ko, but
I have the same perf problems with both solutions. Also I tried
disabling net.inet.tcp.tso and did ifconfig -tso on all NICs on NAT
gateway and other VMs without change.

I don't see anything on the error console or in /var/log/message.

When i'm trying going though NAT with a Linux paravirt DomU, the network
perf is slightly better but don't go over 5KiB/s.

Finally I put my Linux paravirt DomU VM as a NAT gateway in place of the
FreeBSD one, and the network perf is now normal through it (50MiB/s)
from the FreeBSD DomU VMs.

As I'm very happy with FreeBSD for my environment I'd like to keep as
much as this lovely OS for all my needs, so if anyone have an idea where
the problem can lies, and what to test next to have a clue of what's
happening, I will be very thankfull.

Thanks in advance.

-- 
/Regards,/

*Laurent Cligny
/Founder/*
02 97 37 71 14
06 25 40 86 48
laurent.cligny at steadinet.fr

Steadinet <http://www.steadinet.fr>