FreeBSD 10-R, Xen 4.1 guest, pf/NAT performance question

[seanrees at gmail.com]

Hi there freebsd-questions,

I've been batting my head against this problem for a few days now and not
having much progress, so I'm hoping to get pointers at what to look at next.

I've got a FreeBSD 10-R guest in Xen 4.1 (I am just a customer of the Xen
provider; I don't run the Xen hypervisor myself). I use this instance to
terminate a VPN, for which I also NAT VPN clients with PF. I am seeing
unusually slow packet forwarding performance: 0.5mbit internet -> vpn
client, 2.0 mbit vpn client -> internet. (the numbers should be closer to
10mbit/5mbit).

This guest is a duplicate of another Xen instance I have in another data
centre. I manage the configurations and packages centrally and aside from
IP address differences, the machines are configured identically. The
differences: it's 30ms closer to me and runs in Xen 3.4. I see performance
from this machine in the 10mbps range.

I've eliminated the obvious:
  - The problem VPS is fine network wise; can download tarballs from the
Internet at 100mbps.
  - VPS -> Home is fine; can download at ~10mbps; the problem is isolated
to forwarding Home -> VPS -> Internet and back.
  - I excluded OpenVPN as the cause by replicating the setup with ssh -w;
same performance.
  - SSH port forwarding (ssh -L) is fast; indicating to me the issue is
somewhere in the PF/kernel.
  - I checked TCP options by capturing traffic at varying points; these
seem fine. I see a good deal of TCP retransmits but the window sizes stay
the same.

Any thoughts on what to check next?

Thanks,
Sean