1gbit LFN WAN link - odd tcp behavior

[William Salt]

Hi All,
         For the last couple of months i have been pulling my hair out
trying to solve this problem.
We have a 1Gbps transatlantic link from the UK to the US, which has
successfully passed the RFC2544 test.

At either end, we have a media converter, and a supermicro server with an
intel quad port NIC running pfsense 2 (freebsd 8.1) with the NIC running on
the yandex IGB driver.

We can pass 1gbps either way with UDP. However we are experiencing very
strange issues with tcp connections.

With window scaling enabled, and a max socket buffer set to 16MB, we see no
difference.
Even disabling window scaling and setting the window to 16MB makes no
difference.

Each TCP connection starts very slowly, and will max out at around 190mbps,
taking nearly 2 minutes to climb to this speed before *plateauing*.

We have to initiate many (5+) connections to saturate the link with tcp
connections with iperf.

I have followed guides like this:
http://www.psc.edu/networking/projects/tcptune/#FreeBSD

With no luck, and have tweaked, disabled, and enabled nearly every relevant
sysctl parameter with no luck.

Can anyone shed some light on this?

I am now doubting the IGB driver, and am looking to swap out the cards as a
last ditch effort.
However, we have tried different hardware (L3 switches, media convertes +
laptops etc), and the symptoms still persist...
The only constant is freebsd 8.1 (or 8.2 for production systems).


Cheers in advance
Will