Packets delays
Дмуха Николай
cosmic17 at yandex.ru
Tue Nov 3 20:41:50 UTC 2009
Hello!
We have a problem...
Dlink 3426 is core of our network. DGS 3612 - routers, which are connected to core (users). Core is connected to bridge. There are three interfaces igb0, igb1, igb2 on bridge:
igb0 - local network
igb1 - border1
igb2 - border2
Bridge is are shaper (ipfw). border1 and border2 are connected to bridge.
Border1:
FreeBSD 7.0-STABLE-200807 i386
Border2:
FreeBSD 7.2-STABLE-200906 i386
Bridge:
FreeBSD 7.2-STABLE-200906 amd64
Traffic is devided between two borders. We have not got any problems with border2. But on border1 we have: ping through border1 from users and from bridge
ping ya.ru
PING ya.ru (93.158.134.8): 56 data bytes
64 bytes from 93.158.134.8: icmp_seq=0 ttl=58 time=95.083 ms
64 bytes from 93.158.134.8: icmp_seq=1 ttl=58 time=102.876 ms
64 bytes from 93.158.134.8: icmp_seq=2 ttl=58 time=113.630 ms
64 bytes from 93.158.134.8: icmp_seq=3 ttl=58 time=90.352 ms
64 bytes from 93.158.134.8: icmp_seq=4 ttl=58 time=127.349 ms
64 bytes from 93.158.134.8: icmp_seq=5 ttl=58 time=115.942 ms
64 bytes from 93.158.134.8: icmp_seq=6 ttl=58 time=103.753 ms
64 bytes from 93.158.134.8: icmp_seq=7 ttl=58 time=124.507 ms
But in border1 we have:
ping ya.ru
PING ya.ru (213.180.204.8): 56 data bytes
64 bytes from 213.180.204.8: icmp_seq=0 ttl=61 time=3.130 ms
64 bytes from 213.180.204.8: icmp_seq=1 ttl=61 time=2.909 ms
64 bytes from 213.180.204.8: icmp_seq=2 ttl=61 time=3.791 ms
64 bytes from 213.180.204.8: icmp_seq=3 ttl=61 time=2.953 ms
64 bytes from 213.180.204.8: icmp_seq=4 ttl=61 time=3.298 ms
64 bytes from 213.180.204.8: icmp_seq=5 ttl=61 time=3.096 ms
bridge:
1).
netstat -w1d -I igb0
input (igb0) output
packets errs bytes packets errs bytes colls
32442 0 19375201 29681 0 23220291 0
31894 0 19334182 29124 0 23043942 0
31566 0 18885017 28558 0 22390016 0
31810 0 19200993 28768 0 22270687 0
31879 0 19347245 29145 0 22795440 0
31697 0 18997706 29274 0 22651927 0
32042 0 18963695 29634 0 23508103 0
30674 0 18123997 28432 0 22890282 0
31654 0 18519433 28860 0 22918799 0
31961 0 19145696 29418 0 23238031 0
32056 0 19053994 29723 0 23775215 0
32367 0 18952231 29771 0 23774054 0
2).
netstat -w1d -I igb1
input (igb1) output
packets errs bytes packets errs bytes colls
21378 0 19903174 21410 0 9866489 0
21444 0 19948761 21752 0 10266869 0
21435 0 19972984 21825 0 10289265 0
21318 0 19748715 21431 0 10238696 0
21606 0 19855988 21467 0 10483023 0
21825 0 19956737 21750 0 10452641 0
21665 0 19805016 21980 0 10711357 0
20937 0 18983521 21357 0 10506764 0
21241 0 19280484 21992 0 10713008 0
21469 0 19652542 21667 0 10391424 0
21235 0 19538130 21524 0 10757572 0
20779 0 18963762 21003 0 10525245 0
20853 0 18854491 21312 0 10559890 0
Those commands on border1:
1). internal
netstat -w1d -I em1
input (em1) output
packets errs bytes packets errs bytes colls
9862 0 8286463 8226 0 4172860 0
10060 0 8603460 7209 0 3184874 0
9290 0 7781648 7709 0 3672720 0
9720 0 8311391 7208 0 3545028 0
10256 0 8669129 8232 0 4678531 0
9509 0 8140150 7715 0 3878637 0
9554 0 8081766 7199 0 3512477 0
9426 0 8046217 7721 0 3867273 0
9795 0 8177803 7721 0 3729722 0
9637 0 8297993 7199 0 3766607 0
9671 0 8275210 7202 0 3510855 0
9451 0 8306122 7196 0 3535604 0
9324 0 8009044 7202 0 3092661 0
9715 0 8265237 7364 0 3575433 0
9908 0 8215978 7499 0 3911766 0
9776 0 8270432 8086 0 4178572 0
2). external
netstat -w1d -I em0
input (em0) output
packets errs bytes packets errs bytes colls
8253 0 4640677 9803 0 8495567 0
7897 0 4878726 10041 0 7322200 0
8368 0 4776220 9832 0 8565609 0
7650 0 4285985 9283 0 7717595 0
7891 0 4352473 9797 0 8134889 0
8606 0 5018218 9657 0 8289548 0
8460 0 4720658 10207 0 8397786 0
7543 0 4085187 9277 0 8059337 0
9376 0 6040976 11500 0 8860064 0
8181 0 4552212 10295 0 8582838 0
8774 0 5079266 10062 0 8494193 0
8193 0 4674008 10295 0 8520010 0
10170 0 7212068 10813 0 8596957 0
8499 0 4936674 10828 0 9090783 0
8416 0 4409607 10297 0 9050824 0
7613 0 3960242 9280 0 8231506 0
A few month ago border1 in this configuration worked succsessfully..
On border1 there is yandex driver for em.
loader.conf:
hw.em.rxd="4096"
hw.em.txd="4096"
/etc/sysctl.conf на border1:
net.inet.tcp.recvspace=262144
net.inet.tcp.sendspace=62144
net.inet.udp.recvspace=1048576
kern.ipc.maxsockbuf=4194304
kern.ipc.nmbclusters=262144
net.inet.ip.portrange.first=5700
kern.ipc.somaxconn=65535
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.icmplim=30
net.inet.ip.intr_queue_maxlen=4096
net.inet.tcp.delayed_ack=0
net.inet.tcp.delacktime=10
net.inet.tcp.newreno=0
net.inet.tcp.msl=2500
net.inet.ip.rtmaxcache=1024
net.inet.raw.recvspace=65536
net.inet.ip.dummynet.hash_size=65536
net.inet.ip.fw.dyn_ack_lifetime=60
net.inet.ip.fw.dyn_syn_lifetime=10
net.inet.ip.fw.dyn_fin_lifetime=10
net.inet.ip.fw.dyn_max=16192
net.inet.ip.fastforwarding=1
net.isr.direct=1
net.inet.icmp.drop_redirect=1
dev.em.0.rx_kthreads=4
dev.em.1.rx_kthreads=4
dev.em.2.rx_kthreads=4
I have made the same options in sysctl on border1 like on border2:
kern.ipc.nmbclusters=262144
net.inet.icmp.icmplim=200
I have added a new option
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.bpf.bufsize=4194304
net.bpf.maxbufsize=8388608
But have no ane good results.
dmesg:
Border2 external:
em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 0
em0: Receive No Buffers = 0
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 0
em0: watchdog timeouts = 0
em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 5018002561
em0: Good Packets Xmtd = 5133609771
em0: TSO Contexts Xmtd = 0
em0: TSO Contexts Failed = 0
Border1 internal:
em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 9975
em0: Receive No Buffers = 7099
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 1
em0: watchdog timeouts = 0
em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 2639750787
em0: Good Packets Xmtd = 2455117624
em0: TSO Contexts Xmtd = 18
em0: TSO Contexts Failed = 0
Border2 internal:
em1: Excessive collisions = 0
em1: Sequence errors = 0
em1: Defer count = 0
em1: Missed Packets = 0
em1: Receive No Buffers = 0
em1: Receive Length Errors = 0
em1: Receive errors = 0
em1: Crc errors = 0
em1: Alignment errors = 0
em1: Collision/Carrier extension errors = 0
em1: RX overruns = 0
em1: watchdog timeouts = 0
em1: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em1: XON Rcvd = 0
em1: XON Xmtd = 0
em1: XOFF Rcvd = 0
em1: XOFF Xmtd = 0
em1: Good Packets Rcvd = 5189211824
em1: Good Packets Xmtd = 4995412107
em1: TSO Contexts Xmtd = 16447
em1: TSO Contexts Failed = 0
Border1 internal:
em1: Excessive collisions = 0
em1: Sequence errors = 0
em1: Defer count = 0
em1: Missed Packets = 0
em1: Receive No Buffers = 0
em1: Receive Length Errors = 0
em1: Receive errors = 0
em1: Crc errors = 0
em1: Alignment errors = 0
em1: Collision/Carrier extension errors = 0
em1: RX overruns = 0
em1: watchdog timeouts = 0
em1: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em1: XON Rcvd = 0
em1: XON Xmtd = 0
em1: XOFF Rcvd = 0
em1: XOFF Xmtd = 0
em1: Good Packets Rcvd = 2469872680
em1: Good Packets Xmtd = 2578357516
em1: TSO Contexts Xmtd = 14174
em1: TSO Contexts Failed = 0
Bridge:
internal
igb0: Adapter hardware address = 0xffffff00012f0528
igb0: CTRL = 0xc00241 RCTL = 0x801a
igb0: Packet buffer = Tx=0k Rx=0k
igb0: Flow control watermarks high = 63488 low = 61988
igb0: Queue(0) tdh = 154, tdt = 154
igb0: no descriptors avail event = 0
igb0: TX(0) MSIX IRQ Handled = 26582786007
igb0: TX(0) Packets sent = 69466117227
igb0: Queue(1) tdh = 0, tdt = 0
igb0: no descriptors avail event = 0
igb0: TX(1) MSIX IRQ Handled = 0
igb0: TX(1) Packets sent = 0
igb0: Queue(2) tdh = 0, tdt = 0
igb0: no descriptors avail event = 0
igb0: TX(2) MSIX IRQ Handled = 0
igb0: TX(2) Packets sent = 0
igb0: Queue(3) tdh = 0, tdt = 0
igb0: no descriptors avail event = 0
igb0: TX(3) MSIX IRQ Handled = 0
igb0: TX(3) Packets sent = 0
igb0: Queue(0) rdh = 186, rdt = 185
igb0: RX(0) Packets received = 18340736258
igb0: RX(0) Split Packets = 0
igb0: RX(0) Byte count = 12477314664354
igb0: RX(0) MSIX IRQ Handled = 11343806385
igb0: RX(0) LRO Queued= 0
igb0: RX(0) LRO Flushed= 0
igb0: Queue(1) rdh = 150, rdt = 149
igb0: RX(1) Packets received = 18360157016
igb0: RX(1) Split Packets = 0
igb0: RX(1) Byte count = 12474408387147
igb0: RX(1) MSIX IRQ Handled = 11362784395
igb0: RX(1) LRO Queued= 0
igb0: RX(1) LRO Flushed= 0
igb0: Queue(2) rdh = 142, rdt = 140
igb0: RX(2) Packets received = 18122235444
igb0: RX(2) Split Packets = 0
igb0: RX(2) Byte count = 12331139081653
igb0: RX(2) MSIX IRQ Handled = 11268633644
igb0: RX(2) LRO Queued= 0
igb0: RX(2) LRO Flushed= 0
igb0: Queue(3) rdh = 95, rdt = 94
igb0: RX(3) Packets received = 18379433903
igb0: RX(3) Split Packets = 0
igb0: RX(3) Byte count = 12407828772109
igb0: RX(3) MSIX IRQ Handled = 11387935869
igb0: RX(3) LRO Queued= 0
igb0: RX(3) LRO Flushed= 0
igb0: LINK MSIX IRQ Handled = 5
igb0: Mbuf defrag failed = 0
igb0: Std mbuf header failed = 0
igb0: Std mbuf packet failed = 0
igb0: Driver dropped packets = 0
igb0: Driver tx dma failure in xmit = 0
border1
igb1: Adapter hardware address = 0xffffff000451a528
igb1: CTRL = 0x18c00241 RCTL = 0x801a
igb1: Packet buffer = Tx=0k Rx=0k
igb1: Flow control watermarks high = 63488 low = 61988
igb1: Queue(0) tdh = 150, tdt = 151
igb1: no descriptors avail event = 1
igb1: TX(0) MSIX IRQ Handled = 9053729033
igb1: TX(0) Packets sent = 15117561092
igb1: Queue(1) tdh = 0, tdt = 0
igb1: no descriptors avail event = 0
igb1: TX(1) MSIX IRQ Handled = 0
igb1: TX(1) Packets sent = 0
igb1: Queue(2) tdh = 0, tdt = 0
igb1: no descriptors avail event = 0
igb1: TX(2) MSIX IRQ Handled = 0
igb1: TX(2) Packets sent = 0
igb1: Queue(3) tdh = 0, tdt = 0
igb1: no descriptors avail event = 0
igb1: TX(3) MSIX IRQ Handled = 0
igb1: TX(3) Packets sent = 0
igb1: Queue(0) rdh = 253, rdt = 251
igb1: RX(0) Packets received = 3887801598
igb1: RX(0) Split Packets = 0
igb1: RX(0) Byte count = 3419307814950
igb1: RX(0) MSIX IRQ Handled = 2880414195
igb1: RX(0) LRO Queued= 0
igb1: RX(0) LRO Flushed= 0
igb1: Queue(1) rdh = 119, rdt = 118
igb1: RX(1) Packets received = 3908297847
igb1: RX(1) Split Packets = 0
igb1: RX(1) Byte count = 3460826092933
igb1: RX(1) MSIX IRQ Handled = 2888114773
igb1: RX(1) LRO Queued= 0
igb1: RX(1) LRO Flushed= 0
igb1: Queue(2) rdh = 152, rdt = 151
igb1: RX(2) Packets received = 3922202264
igb1: RX(2) Split Packets = 0
igb1: RX(2) Byte count = 3503326896463
igb1: RX(2) MSIX IRQ Handled = 2886529770
igb1: RX(2) LRO Queued= 0
igb1: RX(2) LRO Flushed= 0
igb1: Queue(3) rdh = 92, rdt = 91
igb1: RX(3) Packets received = 3887004765
igb1: RX(3) Split Packets = 0
igb1: RX(3) Byte count = 3441233186193
igb1: RX(3) MSIX IRQ Handled = 2879712428
igb1: RX(3) LRO Queued= 0
igb1: RX(3) LRO Flushed= 0
igb1: LINK MSIX IRQ Handled = 10
igb1: Mbuf defrag failed = 0
igb1: Std mbuf header failed = 0
igb1: Std mbuf packet failed = 0
igb1: Driver dropped packets = 0
igb1: Driver tx dma failure in xmit = 0
Border2:
igb2: Adapter hardware address = 0xffffff0004575528
igb2: CTRL = 0x18c00241 RCTL = 0x801a
igb2: Packet buffer = Tx=0k Rx=0k
igb2: Flow control watermarks high = 63488 low = 61988
igb2: Queue(0) tdh = 74, tdt = 74
igb2: no descriptors avail event = 3117
igb2: TX(0) MSIX IRQ Handled = 22305812305
igb2: TX(0) Packets sent = 57738555533
igb2: Queue(1) tdh = 0, tdt = 0
igb2: no descriptors avail event = 0
igb2: TX(1) MSIX IRQ Handled = 0
igb2: TX(1) Packets sent = 0
igb2: Queue(2) tdh = 0, tdt = 0
igb2: no descriptors avail event = 0
igb2: TX(2) MSIX IRQ Handled = 0
igb2: TX(2) Packets sent = 0
igb2: Queue(3) tdh = 0, tdt = 0
igb2: no descriptors avail event = 0
igb2: TX(3) MSIX IRQ Handled = 0
igb2: TX(3) Packets sent = 0
igb2: Queue(0) rdh = 246, rdt = 245
igb2: RX(0) Packets received = 13727912438
igb2: RX(0) Split Packets = 0
igb2: RX(0) Byte count = 10677902707967
igb2: RX(0) MSIX IRQ Handled = 8730210616
igb2: RX(0) LRO Queued= 0
igb2: RX(0) LRO Flushed= 0
igb2: Queue(1) rdh = 51, rdt = 50
igb2: RX(1) Packets received = 13676394291
igb2: RX(1) Split Packets = 0
igb2: RX(1) Byte count = 10622417591526
igb2: RX(1) MSIX IRQ Handled = 8706454649
igb2: RX(1) LRO Queued= 0
igb2: RX(1) LRO Flushed= 0
igb2: Queue(2) rdh = 194, rdt = 193
igb2: RX(2) Packets received = 13687518658
igb2: RX(2) Split Packets = 0
igb2: RX(2) Byte count = 10608204213317
igb2: RX(2) MSIX IRQ Handled = 8716002420
igb2: RX(2) LRO Queued= 0
igb2: RX(2) LRO Flushed= 0
igb2: Queue(3) rdh = 131, rdt = 130
igb2: RX(3) Packets received = 13660077699
igb2: RX(3) Split Packets = 0
igb2: RX(3) Byte count = 10588060857368
igb2: RX(3) MSIX IRQ Handled = 8692141405
igb2: RX(3) LRO Queued= 0
igb2: RX(3) LRO Flushed= 0
igb2: LINK MSIX IRQ Handled = 32
igb2: Mbuf defrag failed = 0
igb2: Std mbuf header failed = 0
igb2: Std mbuf packet failed = 0
igb2: Driver dropped packets = 0
We have changed the cable between bridge and borfer1...but haven`t got any results
More information about the freebsd-questions
mailing list