Significant network latency when using ipfw and in-kernel NAT

Mon Sep 17 12:04:20 UTC 2012

> what about the other one ? Also, please disable jumbo_mtu as well.
> On both inside and outside.

As far as I was able to tell, VLAN_HWCSUM cannot be disabled (or I
don't know which command to use):
http://lists.freebsd.org/pipermail/freebsd-net/2004-March/003464.html

I also don't know how to disable JUMBO_MTU and VLAN_MTU.

Disabling VLAN_HWCSUM didn't seem to do anything. Everything still has
just as much latency as before:

ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=a8<VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM>

Here is the current ruleset:

00001  32195  17958479 allow ip from any to any via ix0
00002      0         0 allow ip from any to any via gif0
00003  14593   1030091 allow ip from any to any via gif1
00004  17210  16260592 allow ip from any to any via gif2
00005      0         0 allow ip from any to any via gif3
00006      0         0 allow ip from any to any via lo0
00015      0         0 deny ip from 192.168.0.0/16 to any in via ix1
00016      0         0 deny ip from 172.16.0.0/12 to any in via ix1
00017      0         0 deny ip from 10.0.0.0/8 to any in via ix1
00018      0         0 deny ip from 127.0.0.0/8 to any in via ix1
00019      0         0 deny ip from 0.0.0.0/8 to any in via ix1
00020      0         0 deny ip from 169.254.0.0/16 to any in via ix1
00021      0         0 deny ip from 192.0.2.0/24 to any in via ix1
00022      0         0 deny ip from 204.152.64.0/23 to any in via ix1
00023      0         0 deny ip from 224.0.0.0/3 to any in via ix1
00025     11      1118 allow icmp from any to any icmptypes 3,11 in recv ix1
00026      6       264 deny icmp from any to any in recv ix1
00040  13121    745760 nat 1 ip from any to any in recv ix1
00050      0         0 check-state
00100     17       924 skipto 805 tcp from any to any out xmit ix1
setup keep-state
00202   5903    293907 skipto 600 tcp from any to 172.16.1.3 dst-port
443 in via ix1
00203  11289  15948611 skipto 805 tcp from 172.16.1.3 443 to any out xmit ix1
00204   7212    451553 skipto 700 tcp from any to 172.16.1.4 dst-port
5222 in via ix1
00205   7377    578378 skipto 805 tcp from 172.16.1.4 5222 to any out xmit ix1
00400     11      3564 deny ip from any to any via ix1
00500      0         0 pipe 1 ip from any to any in via ix1
00501      0         0 allow ip from any to any in via ix1
00600   5902    293361 pipe 2 ip from any to any in via ix1
00601   5902    293361 allow ip from any to any in via ix1
00700   7210    451399 pipe 3 ip from any to any in via ix1
00701   7210    451399 allow ip from any to any in via ix1
00800      0         0 pipe 4 ip from any to any in via ix1
00801      0         0 allow ip from any to any in via ix1
00805  18672  16520573 nat 1 ip from any to any out xmit ix1
00806  18672  16520573 allow ip from any to any
10000      0         0 deny ip from any to any via ix1
65535 865391 867355171 allow ip from any to any

And the pipes:

00001:  XX.000 Mbit/s    0 ms burst 0
q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:  XX.000 Mbit/s    0 ms burst 0
q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active
00003: XX.000 Mbit/s    0 ms burst 0
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active
00004:  XX.000 Mbit/s    0 ms burst 0
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active

Like I mentioned earlier, one-pass is set to 0 to allow for traffic to
be put back in to ipfw after going through NAT'ing and the pipes. That
couldn't affect negatively, right?

Cheers,
Soren

On Sun, Sep 16, 2012 at 11:21 PM, Luigi Rizzo <rizzo at iet.unipi.it> wrote:
> On Sun, Sep 16, 2012 at 10:39:36PM -0500, Soren Dreijer wrote:
>> Some more updates:
>>
>> I went ahead and disabled a few options on the ixgbe network interface
>> today (most notably rxcsum and txcsum), which improved ping times to
>> the FreeBSD box. I'm now able to reliably ping it with ~40ms from my
>> house. TCP traffic in general also seems to be slightly "better" as I
>> can actually 'wget google.com' now, although it's still horribly slow
>> and takes maybe 20 seconds or so to download.
>>
>> The ifconfig for the public adapter now looks like this:
>>
>> ix1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         options=b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
>
> what about the other one ? Also, please disable jumbo_mtu as well.
> On both inside and outside.
>
> Finally, can you send the output of
> "ipfw show" and "ipfw pipe show" (anonymized if you like, but
> please preserve the counters) to see if there is any traffic
> that is looping ?
>
> thanks
> luigi
>
>>
>> I'm running out of ideas of what to do here...
>>
>> / Soren
>>