With several days more data, the Realtek driver is slightly different
than the stock FreeBSD 10.3 driver, but it still fails a lot, with both
TCP and UDP.
> Has anyone tried similar testing?
> Suggestions of knobs to turn, or other things to try?
> I need to receive data via UDP without dropping packets.
> Closed source sucks, but I'm stuck with it, and thus with UDP.
Gary typed:
>> Everyone blames re, but AFAICT
>> no-one switched to a non-Realtek chip to run tests and prove that,
>> yes, re is really the cause of all the network problems.
I have a box with nfe(4) and bge(4) (BCM5750) that can receive data
with UDP or TCP without error. This has been working for several years.
But a newer, faster machine with re(4) 8111E & 8111F cannot. The nfe is
in the chipset, and the BCM5750 is onboard. I had google look for an
expansion card with BCM5750 but found nothing. I'm looking into
a Broadcom 5719 card, but there is at least one open PR against it, so...
Alex typed:
> surely it's not a NFS issue
NFS is a bug. By design. "Not a File System"
My machines are NFS-free.
> The only thing that link all bug reports
> (and there are tons) is the use of re card sustaining gigabit (duplex)
> transfers.
My re run (1000baseT <full-duplex>) but the pauses happen even with
light traffic. You don't have to run them full blast with rcp/ftp/whatever
to get failures.
STefan typed:
> Well, but you know that the U in UDP means unreliable?
Well, :-) you didn't pay attention when I said exactly that:
>>> Something like a rcp(1) with another
>>> FreeBSD box merely runs slower due to the stalls, but if the other end
>>> has buggy network code and/or if the transfer is forced to use UDP
>>> (Unreliable Data Protocol) data is lost. :-(
>> With reasonably decent network software and TCP, this is only a minor
>> problem. However with buggy network software (like a "black box"
>> with closed source firmware and maybe a transmit buffer that is *way*
>> too small, and/or true real time requirements), and/or with a brain dead
>> protocol like UDP, you can lose data.
rcp(1) to 8111F using Realtek driver
Both machines basically idle.
mtu=9000 on both ends but appears to be using 1500 anyway?
netstat -w 1 -d -I re0
input re0 output
packets errs idrops bytes packets errs bytes colls drops
5 0 0 330 6 0 755 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
17454 0 0 26176844 17453 0 1152264 0 0
67883 0 0 101023187 67880 0 4481248 0 0
52133 0 0 77141014 52140 0 3444580 0 0
72410 0 0 107980584 72409 0 4780384 0 0
80736 0 0 120645177 80735 0 5328810 0 0
71898 0 0 107298972 71899 0 4745634 0 0
14452 0 0 21510681 14475 0 955577 0 0
5 0 0 330 6 0 755 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
4 0 0 264 4 0 564 0 0
65323 0 0 97533279 65321 0 4311552 0 0
28383 0 0 42398766 28385 0 1873769 0 0
80759 0 0 120315366 80756 0 5330394 0 0
80992 0 0 120693553 80995 0 5345772 0 0
80932 0 0 120692488 80930 0 5341746 0 0
78905 0 0 117912275 78922 0 5209152 0 0
80158 0 0 119592452 80159 0 5290728 0 0
input re0 output
packets errs idrops bytes packets errs bytes colls drops
80851 0 0 120590446 80852 0 5336591 0 0
80849 0 0 120589243 80847 0 5336334 0 0
80858 0 0 120598524 80860 0 5336928 0 0
15085 0 0 22529747 15107 0 997296 0 0
4 0 0 264 4 0 564 0 0
4 0 0 264 4 0 564 0 0
5 0 0 331 5 0 630 0 0
4 0 0 264 4 0 564 0 0
63944 0 0 95447185 63943 0 4220604 0 0
80588 0 0 120276192 80589 0 5319233 0 0
80721 0 0 120480130 80722 0 5328011 0 0
79870 0 0 119139685 79870 0 5271720 0 0
79425 0 0 118587402 79442 0 5243472 0 0
80609 0 0 120333747 80609 0 5320494 0 0
80852 0 0 120624944 80850 0 5336532 0 0
80827 0 0 120555558 80831 0 5335073 0 0
79185 0 0 118137499 79185 0 5226510 0 0
76417 0 0 114252562 76431 0 5045010 0 0
79896 0 0 119330393 79900 0 5273436 0 0
79455 0 0 118627478 79456 0 5244330 0 0
77667 0 0 116033902 77662 0 5126256 0 0
input re0 output
packets errs idrops bytes packets errs bytes colls drops
79517 0 0 118735507 79515 0 5248547 0 0
78557 0 0 117304898 78578 0 5186118 0 0
79607 0 0 118742143 79625 0 5255616 0 0
79776 0 0 119140720 79775 0 5265450 0 0
75105 0 0 112725906 75107 0 4957230 0 0
79214 0 0 118774181 79214 0 5228424 0 0
13090 0 0 19520508 13113 0 865692 0 0
Same as above except to 2nd 8111F port.
input re1 output
packets errs idrops bytes packets errs bytes colls drops
75170 0 0 112288950 75165 0 4960956 0 0
74149 0 0 110808698 74150 0 4893834 0 0
75088 0 0 112276960 75088 0 4955808 0 0
74567 0 0 111313160 74566 0 4921356 0 0
77064 0 0 115370280 77065 0 5086290 0 0
71606 0 0 107343414 71622 0 4727052 0 0
70831 0 0 105956278 70832 0 4674849 0 0
70727 0 0 106127094 70725 0 4667916 0 0
73427 0 0 110111478 73425 0 4846182 0 0
72519 0 0 108763678 72522 0 4786320 0 0
5661 0 0 8423884 5681 0 374880 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
2 0 0 196 0 0 0 0 0
1 0 0 98 0 0 0 0 0
563 0 0 838984 561 0 37026 0 0
29144 0 0 43754296 29142 0 1923438 0 0
73986 0 0 110909436 73987 0 4883142 0 0
39601 0 0 59343834 39621 0 2614920 0 0
0 0 0 0 0 0 0 0 0
3 0 0 201 0 0 0 0 0
input re1 output
packets errs idrops bytes packets errs bytes colls drops
0 0 0 0 0 0 0 0 0
1 0 0 98 0 0 0 0 0
567 0 0 839958 567 0 37422 0 0
1 0 0 98 0 0 0 0 0
38003 0 0 56780696 38000 0 2508066 0 0
75439 0 0 112523550 75438 0 4978908 0 0
77058 0 0 114890108 77059 0 5085894 0 0
77471 0 0 115473190 77470 0 5113020 0 0
76814 0 0 114644092 76815 0 5069790 0 0
78120 0 0 116633730 78118 0 5155788 0 0
74853 0 0 111727082 74854 0 4940298 0 0
77626 0 0 115697756 77623 0 5123250 0 0
70804 0 0 105582200 70804 0 4672998 0 0
76722 0 0 114492148 76722 0 5063652 0 0
29562 0 0 44157654 29582 0 1952346 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
568 0 0 840298 566 0 37356 0 0
0 0 0 0 0 0 0 0 0
input re1 output
packets errs idrops bytes packets errs bytes colls drops
43130 0 0 64590180 43128 0 2846514 0 0
29010 0 0 43374204 29032 0 1916046 0 0
0 0 0 0 0 0 0 0 0
2 0 0 134 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
515 0 0 770942 515 0 33990 0 0
17903 0 0 26853958 17902 0 1181598 0 0
74854 0 0 111699988 74851 0 4940166 0 0
77120 0 0 115061536 77117 0 5089722 0 0
76857 0 0 114664362 76855 0 5072562 0 0
75817 0 0 113174146 75820 0 5003988 0 0
76737 0 0 114435914 76737 0 5064642 0 0
78495 0 0 117128814 78493 0 5180604 0 0
75723 0 0 112970948 75724 0 4997718 0 0
76501 0 0 114196604 76500 0 5049066 0 0
76187 0 0 113779382 76189 0 5028408 0 0
77236 0 0 115269384 77235 0 5097510 0 0
75300 0 0 112389128 75300 0 4969800 0 0
76492 0 0 114455568 76492 0 5048472 0 0
75573 0 0 113011418 75572 0 4987752 0 0
input re1 output
packets errs idrops bytes packets errs bytes colls drops
73331 0 0 109541752 73330 0 4839780 0 0
76732 0 0 114631816 76733 0 5064378 0 0
74400 0 0 111067456 74399 0 4910334 0 0
75974 0 0 113463012 75975 0 5014350 0 0
76312 0 0 113999448 76311 0 5036526 0 0
43806 0 0 65415884 43844 0 2893638 0 0
1 0 0 66 0 0 0 0 0
0 0 0 0 0 0 0 0 0
Same as above except to onboard 8111E.
input re2 output
packets errs idrops bytes packets errs bytes colls drops
53122 0 0 80345556 0 0 0 0 0
53984 0 0 81332624 0 0 0 0 0
61394 0 0 92785548 0 0 0 0 0
53421 0 0 80696090 0 0 0 0 0
30857 0 0 46597698 0 0 0 0 0
76124 0 0 115239504 0 0 0 0 0
69172 0 0 104706904 0 0 0 0 0
76094 0 0 115191956 0 0 0 0 0
76849 0 0 116329874 0 0 0 0 0
68788 0 0 104120096 0 0 0 0 0
69233 0 0 104802146 0 0 0 0 0
50527 0 0 76465702 0 0 0 0 0
75916 0 0 114920192 0 0 0 0 0
66062 0 0 99990092 0 0 0 0 0
76257 0 0 115436194 0 0 0 0 0
47074 0 0 71233028 0 0 0 0 0
51788 0 0 78359104 0 0 0 0 0
44486 0 0 67316276 0 0 0 0 0
76159 0 0 115288566 0 0 0 0 0
64766 0 0 98037852 0 0 0 0 0
76786 0 0 116237604 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
50676 0 0 76688424 0 0 0 0 0
76918 0 0 116440932 0 0 0 0 0
65723 0 0 99488694 0 0 0 0 0
73756 0 0 111653288 0 0 0 0 0
71393 0 0 108070674 0 0 0 0 0
63089 0 0 95485346 0 0 0 0 0
69588 0 0 105333248 0 0 0 0 0
59215 0 0 89617670 0 0 0 0 0
57339 0 0 86752950 0 0 0 0 0
75739 0 0 114651942 0 0 0 0 0
68442 0 0 103594692 0 0 0 0 0
49990 0 0 75649100 0 0 0 0 0
27710 0 0 41940348 0 0 0 0 0
50005 0 0 75694058 0 0 0 0 0
76748 0 0 116183384 0 0 0 0 0
52088 0 0 78847168 0 0 0 0 0
76816 0 0 116287048 0 0 0 0 0
70232 0 0 106301624 0 0 0 0 0
76467 0 0 115754502 0 0 0 0 0
68979 0 0 104412918 0 0 0 0 0
76694 0 0 116101068 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
68857 0 0 104224114 0 0 0 0 0
76545 0 0 115871154 0 0 0 0 0
65996 0 0 99843912 0 0 0 0 0
76674 0 0 116070772 0 0 0 0 0
68095 0 0 103070230 0 0 0 0 0
74877 0 0 113319642 0 0 0 0 0
39228 0 0 59366824 0 0 0 0 0
53026 0 0 80203068 0 0 0 0 0
...
77799 0 0 117779926 0 0 0 0 0
72666 0 0 110007220 0 0 0 0 0
69173 0 0 104681578 0 0 0 0 0
55918 0 0 84169172 0 0 0 0 0
24127 0 0 36195390 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
0 0 0 0 0 0 0 0 0
19706 0 0 29578468 0 0 0 0 0
53262 0 0 79974084 0 0 0 0 0
59033 0 0 88835122 0 0 0 0 0
54469 0 0 81830514 0 0 0 0 0
55005 0 0 82842706 0 0 0 0 0
47867 0 0 72331934 0 0 0 0 0
65468 0 0 99099624 0 0 0 0 0
80572 0 0 121980632 0 0 0 0 0
58201 0 0 88091802 0 0 0 0 0
51948 0 0 78614144 0 0 0 0 0
80053 0 0 121196242 0 0 0 0 0
52331 0 0 79200470 0 0 0 0 0
80123 0 0 121302582 0 0 0 0 0
43414 0 0 65698001 0 0 0 0 0
77961 0 0 117995130 0 0 0 0 0
62287 0 0 94226926 0 0 0 0 0
75558 0 0 114290892 0 0 0 0 0
64228 0 0 97158320 0 0 0 0 0
59981 0 0 90654186 0 0 0 0 0
43520 0 0 65739096 0 0 0 0 0
input re2 output
packets errs idrops bytes packets errs bytes colls drops
56570 0 0 85335700 0 0 0 0 0
50066 0 0 75384196 0 0 0 0 0
55959 0 0 84239934 0 0 0 0 0
53192 0 0 80101232 0 0 0 0 0
49002 0 0 73982980 0 0 0 0 0
49713 0 0 74993370 0 0 0 0 0
43984 0 0 66011800 0 0 0 0 0
54324 0 0 81935448 0 0 0 0 0
48502 0 0 72734828 0 0 0 0 0
26191 0 0 39411806 0 0 0 0 0
0 0 0 0 0 0 0 0 0
42582 0 0 64021036 0 0 0 0 0
47216 0 0 70880104 0 0 0 0 0
52239 0 0 78498398 0 0 0 0 0
52517 0 0 79279690 0 0 0 0 0
44212 0 0 66384032 0 0 0 0 0
57225 0 0 86126642 0 0 0 0 0
47902 0 0 72238300 0 0 0 0 0
44933 0 0 67508656 0 0 0 0 0
54967 0 0 82411976 0 0 0 0 0
50724 0 0 76288600 0 0 0 0 0
...
input re2 output
packets errs idrops bytes packets errs bytes colls drops
56426 0 0 84726988 0 0 0 0 0
53332 0 0 79840392 0 0 0 0 0
52763 0 0 78943630 0 0 0 0 0
52190 0 0 78016892 0 0 0 0 0
56889 0 0 85597554 0 0 0 0 0
55265 0 0 82915642 0 0 0 0 0
52162 0 0 78061690 0 0 0 0 0
58041 0 0 87326532 0 0 0 0 0
53244 0 0 79740088 0 0 0 0 0
51387 0 0 76791862 0 0 0 0 0
54187 0 0 81106878 0 0 0 0 0
51846 0 0 77541604 0 0 0 0 0
42316 0 0 63282584 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
590 0 0 890908 0 0 0 0 0
40253 0 0 60411290 0 0 0 0 0
56653 0 0 85264434 0 0 0 0 0
53499 0 0 80125838 0 0 0 0 0
[ ... ]
input re2 output
packets errs idrops bytes packets errs bytes colls drops
52839 0 0 78935238 0 0 0 0 0
53098 0 0 79307900 0 0 0 0 0
49559 0 0 74231830 0 0 0 0 0
79905 0 0 120972890 0 0 0 0 0
66836 0 0 101174008 0 0 0 0 0
75308 0 0 113998016 0 0 0 0 0
47258 0 0 71543932 0 0 0 0 0
32 0 0 48448 0 0 0 0 0
70026 0 0 106002764 0 0 0 0 0
63937 0 0 96604786 0 0 0 0 0
60947 0 0 92062662 0 0 0 0 0
63772 0 0 96410208 0 0 0 0 0
58129 0 0 87582154 0 0 0 0 0
59204 0 0 89190072 0 0 0 0 0
61375 0 0 92741230 0 0 0 0 0
69159 0 0 104622734 0 0 0 0 0
75748 0 0 114661800 0 0 0 0 0
75478 0 0 114253980 0 0 0 0 0
76814 0 0 116282228 0 0 0 0 0
76931 0 0 116458670 0 0 0 0 0
75299 0 0 113979718 0 0 0 0 0