VM in Xen 4.1; poor packet forwarding performance on xn0

[seanrees at gmail.com]

Hi there freebsd-xen,

I tried first on freebsd-questions@ without success, so I thought to retry
here.

I run OpenVPN on a FreeBSD 10.0-Rp1 VM in Xen 4.1 (HVM). I am experiencing
slow network performance on xn0 that seems to have developed after I
upgraded to FreeBSD 10 (no other changes). I can only achieve about 0.5mbps
through this interface when forwarding packets; packets in a single
direction are fine (e.g; downloading to the VPS or pushing from the VPS)
and clock in at many (>10 usually) mbps.

Interestingly, my identical VM (configuration managed centrally) running on
Xen 3.4 (HVM) does *not* have this issue.

I did a little debugging and here's what I've noticed:
   - Not related to OpenVPN, repro'd using ssh -d.
   - Slow VM has a very low rate of context switches (~250) while
forwarding, fast VM has a lot more (~2000) sampled over 5 seconds using
systat -v.
  - I can't repro a context switch limit (tried a limited fork() bomb).
  - Tried with *and* without LRO and TSO on xn0 (and all combinations of
LRO and TSO on/off)

I started digging into the Xen drivers but I realised I am not equipped to
debug this. I'm looking for suggestions on what to check. My provider is
offering to move me to an older Xen host but I'd like to see if I can help
identify a bug either in my configuration or perhaps in FreeBSD's Xen code.

Sean