freebsd-arm Digest, Vol 740, Issue 7 (Rock64 Ethernet testing)
Mark Millard
marklmi at yahoo.com
Thu Jul 9 07:22:46 UTC 2020
On 2020-Jul-8, at 00:27, Mark Millard <marklmi at yahoo.com> wrote:
> On 2020-Jul-7, at 21:52, Mark Millard <marklmi at yahoo.com> wrote:
>
>> On 2020-Jul-7, at 19:12, Oleksandr Tymoshenko <gonzo at bluezbox.com> wrote:
>>
>>> Mark Millard (marklmi at yahoo.com) wrote:
>>>> Any chance that the delays (or other parameters) depend
>>>> on the operating temperature(s) of some parts?
>>>>
>>>> If yes, then some of the following about the Rock64 V2
>>>> context that I have access to might be relevant to
>>>> explaining my already reported V2 results (not much
>>>> for Retr):
>>>>
>>>> A) The Rock64 has a "case" that is really just a top
>>>> and a bottom with posts: open on all 4 sides.
>>>>
>>>> B) It has a fan blowing down on the board from the
>>>> top.
>>>>
>>>> C) It has a heat sink on the SOC, which the fan blows
>>>> on directly.
>>>>
>>>> D) It has a heat sink on the RAM, which the fan also
>>>> blows on directly.
>>>>
>>>> (I've not dealt with a more modern non-debug kernel
>>>> build yet. It still may be some time before I deal
>>>> with that.)
>>>
>>> Temperature is not likely to be a factor in the delay values.
>>> Rock64 V2 has a known issue with Gigabit ethernet stability:
>>>
>>> https://forum.pine64.org/showthread.php?tid=7545
>>> https://sanisimov.com/2019/08/fixing-rock64-v2-gigabit-ethernet/
>>>
>>> Althought judging by description it's more like an almost complete
>>> network lock-up and not performance degradation.
>>>
>>> I received another board with RK3328 today and will investigate
>>> the issue further.
>>
>> Okay.
>>
>> Looks like I should have copied iperf3 output from the server
>> side as well: somewhat different information. The output was
>> still available from the earlier runs so here it is . . .
>>
>> The modern debug-kernel runs:
>>
>> Accepted connection from 192.168.1.109, port 47111
>> [ 5] local 192.168.1.122 port 5201 connected to 192.168.1.109 port 17015
>> [ ID] Interval Transfer Bitrate
>> [ 5] 0.00-1.00 sec 17.5 MBytes 147 Mbits/sec
>> [ 5] 1.00-2.00 sec 45.3 MBytes 380 Mbits/sec
>> [ 5] 2.00-3.00 sec 44.9 MBytes 376 Mbits/sec
>> [ 5] 3.00-4.00 sec 45.2 MBytes 379 Mbits/sec
>> [ 5] 4.00-5.00 sec 44.9 MBytes 377 Mbits/sec
>> [ 5] 5.00-6.00 sec 45.1 MBytes 379 Mbits/sec
>> [ 5] 6.00-7.00 sec 44.5 MBytes 373 Mbits/sec
>> [ 5] 7.00-8.00 sec 45.0 MBytes 378 Mbits/sec
>> [ 5] 8.00-9.00 sec 44.9 MBytes 377 Mbits/sec
>> [ 5] 9.00-10.00 sec 44.5 MBytes 373 Mbits/sec
>> [ 5] 10.00-10.62 sec 27.9 MBytes 379 Mbits/sec
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval Transfer Bitrate
>> [ 5] 0.00-10.62 sec 450 MBytes 355 Mbits/sec receiver
>>
>> Accepted connection from 192.168.1.109, port 22375
>> [ 5] local 192.168.1.122 port 5201 connected to 192.168.1.109 port 54738
>> [ ID] Interval Transfer Bitrate Retr Cwnd
>> [ 5] 0.00-1.00 sec 24.7 MBytes 207 Mbits/sec 0 265 KBytes
>> [ 5] 1.00-2.00 sec 61.6 MBytes 517 Mbits/sec 4 211 KBytes
>> [ 5] 2.00-3.00 sec 61.4 MBytes 515 Mbits/sec 1 352 KBytes
>> [ 5] 3.00-4.00 sec 61.3 MBytes 514 Mbits/sec 4 269 KBytes
>> [ 5] 4.00-5.00 sec 61.4 MBytes 515 Mbits/sec 2 355 KBytes
>> [ 5] 5.00-6.00 sec 61.3 MBytes 514 Mbits/sec 3 304 KBytes
>> [ 5] 6.00-7.00 sec 61.3 MBytes 514 Mbits/sec 2 327 KBytes
>> [ 5] 7.00-8.00 sec 61.4 MBytes 515 Mbits/sec 5 278 KBytes
>> [ 5] 8.00-9.00 sec 61.4 MBytes 515 Mbits/sec 2 393 KBytes
>> [ 5] 9.00-10.00 sec 61.4 MBytes 515 Mbits/sec 3 284 KBytes
>> [ 5] 10.00-10.61 sec 37.3 MBytes 514 Mbits/sec 2 282 KBytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval Transfer Bitrate Retr
>> [ 5] 0.00-10.61 sec 614 MBytes 486 Mbits/sec 28 sender
>>
>> (So a fairly consistent Retr rate.)
>>
>> The non-debug head -r360311 kernel runs:
>>
>> Accepted connection from 192.168.1.109, port 46431
>> [ 5] local 192.168.1.122 port 5201 connected to 192.168.1.109 port 39541
>> [ ID] Interval Transfer Bitrate
>> [ 5] 0.00-1.00 sec 50.3 MBytes 422 Mbits/sec
>> [ 5] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec
>> [ 5] 2.00-3.00 sec 72.9 MBytes 611 Mbits/sec
>> [ 5] 3.00-4.00 sec 72.8 MBytes 611 Mbits/sec
>> [ 5] 4.00-5.00 sec 72.9 MBytes 611 Mbits/sec
>> [ 5] 5.00-6.00 sec 72.9 MBytes 611 Mbits/sec
>> [ 5] 6.00-7.00 sec 72.8 MBytes 611 Mbits/sec
>> [ 5] 7.00-8.00 sec 72.8 MBytes 610 Mbits/sec
>> [ 5] 8.00-9.00 sec 72.8 MBytes 610 Mbits/sec
>> [ 5] 9.00-10.00 sec 72.8 MBytes 610 Mbits/sec
>> [ 5] 10.00-10.32 sec 23.4 MBytes 612 Mbits/sec
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval Transfer Bitrate
>> [ 5] 0.00-10.32 sec 729 MBytes 593 Mbits/sec receiver
>>
>> Accepted connection from 192.168.1.109, port 40223
>> [ 5] local 192.168.1.122 port 5201 connected to 192.168.1.109 port 50696
>> [ ID] Interval Transfer Bitrate Retr Cwnd
>> [ 5] 0.00-1.00 sec 78.5 MBytes 659 Mbits/sec 0 480 KBytes
>> [ 5] 1.00-2.00 sec 113 MBytes 945 Mbits/sec 0 747 KBytes
>> [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec 0 940 KBytes
>> [ 5] 3.00-4.00 sec 84.5 MBytes 709 Mbits/sec 52 368 KBytes
>> [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 0 681 KBytes
>> [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 0 889 KBytes
>> [ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec 0 1.03 MBytes
>> [ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 0 1.17 MBytes
>> [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 0 125 KBytes
>> [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 3 586 KBytes
>> [ 5] 10.00-10.31 sec 34.7 MBytes 942 Mbits/sec 0 667 KBytes
>> - - - - - - - - - - - - - - - - - - - - - - - - -
>> [ ID] Interval Transfer Bitrate Retr
>> [ 5] 0.00-10.31 sec 1.07 GBytes 892 Mbits/sec 55 sender
>>
>> So: Larger total Retr than the modern debug kernel case
>> but not a fairly consistent rate of Retr values.
>>
>>
>> I've still not dealt with updating to a modern non-debug
>> environment to test it.
>>
>
> Turns out that I get non-zero Retr values between
> any two machines that I try, even when both are not
> arm at all.
>
> For example, between a PowerMac G5 (2 sockets, 2 cores each,
> powerpc64) and a Threadripper 1950X (amd64), Both are Gbps
> capable . . .
>
> FBSDG5L2# iperf3 -c 192.168.1.120
> Connecting to host 192.168.1.120, port 5201
> [ 5] local 192.168.1.7 port 17239 connected to 192.168.1.120 port 5201
> [ ID] Interval Transfer Bitrate Retr Cwnd
> [ 5] 0.00-1.00 sec 113 MBytes 946 Mbits/sec 0 489 KBytes
> [ 5] 1.00-2.00 sec 113 MBytes 944 Mbits/sec 0 730 KBytes
> [ 5] 2.00-3.00 sec 113 MBytes 945 Mbits/sec 48 1.03 MBytes
> [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 95 52.8 KBytes
> [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 95 21.4 KBytes
> [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec 96 1.04 MBytes
> [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 91 148 KBytes
> [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec 92 388 KBytes
> [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 99 792 KBytes
> [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 93 1.07 MBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate Retr
> [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 709 sender
> [ 5] 0.00-10.79 sec 1.10 GBytes 874 Mbits/sec receiver
>
>
> Accepted connection from 192.168.1.120, port 24400
> [ 5] local 192.168.1.7 port 5201 connected to 192.168.1.120 port 14839
> [ ID] Interval Transfer Bitrate Retr Cwnd
> [ 5] 0.00-1.00 sec 109 MBytes 914 Mbits/sec 84 668 KBytes
> [ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec 102 942 KBytes
> [ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec 101 603 KBytes
> [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec 95 868 KBytes
> [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec 92 135 KBytes
> [ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec 89 989 KBytes
> [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec 95 28.5 KBytes
> [ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec 100 158 KBytes
> [ 5] 8.00-9.00 sec 112 MBytes 941 Mbits/sec 102 20.0 KBytes
> [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec 99 335 KBytes
> [ 5] 10.00-10.00 sec 7.50 KBytes 525 Mbits/sec 0 344 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate Retr
> [ 5] 0.00-10.00 sec 1.09 GBytes 939 Mbits/sec 959 sender
>
> Accepted connection from 192.168.1.7, port 30451
> [ 5] local 192.168.1.120 port 5201 connected to 192.168.1.7 port 28078
> [ ID] Interval Transfer Bitrate Retr Cwnd
> [ 5] 0.00-1.00 sec 38.8 MBytes 326 Mbits/sec 36 1.61 MBytes
> [ 5] 1.00-2.01 sec 80.6 MBytes 667 Mbits/sec 471 817 KBytes
> [ 5] 2.01-3.00 sec 106 MBytes 901 Mbits/sec 114 688 KBytes
> [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 0 893 KBytes
> [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 2 523 KBytes
> [ 5] 5.00-6.00 sec 111 MBytes 932 Mbits/sec 0 774 KBytes
> [ 5] 6.00-7.00 sec 108 MBytes 903 Mbits/sec 361 551 KBytes
> [ 5] 7.00-8.00 sec 96.3 MBytes 808 Mbits/sec 180 473 KBytes
> [ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 0 740 KBytes
> [ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec 0 934 KBytes
> [ 5] 10.00-10.67 sec 74.1 MBytes 933 Mbits/sec 1 456 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate Retr
> [ 5] 0.00-10.67 sec 1.03 GBytes 833 Mbits/sec 1165 sender
>
> Connecting to host 192.168.1.7, port 5201
> [ 5] local 192.168.1.120 port 60431 connected to 192.168.1.7 port 5201
> [ ID] Interval Transfer Bitrate Retr Cwnd
> [ 5] 0.00-1.00 sec 97.8 MBytes 818 Mbits/sec 0 633 KBytes
> [ 5] 1.00-2.00 sec 111 MBytes 934 Mbits/sec 128 649 KBytes
> [ 5] 2.00-3.00 sec 112 MBytes 937 Mbits/sec 0 863 KBytes
> [ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec 2 394 KBytes
> [ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec 0 693 KBytes
> [ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 897 KBytes
> [ 5] 6.00-7.00 sec 111 MBytes 932 Mbits/sec 2 480 KBytes
> [ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 0 746 KBytes
> [ 5] 8.00-9.00 sec 62.3 MBytes 523 Mbits/sec 320 436 KBytes
> [ 5] 9.00-10.00 sec 111 MBytes 934 Mbits/sec 0 717 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bitrate Retr
> [ 5] 0.00-10.00 sec 1.03 GBytes 881 Mbits/sec 452 sender
> [ 5] 0.00-10.01 sec 1.02 GBytes 880 Mbits/sec receiver
>
> I may need to establish a better context or just may be
> limited to Bitrate comparisons instead of looking for
> Retr staying zero.
>
> (Still not at a modern non-debug build for the Rock64.)
The Rock64 V2 is now running head -r363021 with a non-debug
kernel and world:
# uname -apKU
FreeBSD Rock64orRPi4 13.0-CURRENT FreeBSD 13.0-CURRENT #5 r363021M: Wed Jul 8 20:30:01 PDT 2020 markmi at FBSDFHUGE:/usr/obj/cortexA53_clang/arm64.aarch64/usr/src/arm64.aarch64/sys/GENERIC-NODBG arm64 aarch64 1300100 1300100
The patch for extra messages (and a potential bug
fix) has not been applied.
The iperf3 outputs are below.
# iperf3 -c 192.168.1.120
Connecting to host 192.168.1.120, port 5201
[ 5] local 192.168.1.109 port 40422 connected to 192.168.1.120 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 73.3 MBytes 615 Mbits/sec 0 732 KBytes
[ 5] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec 0 732 KBytes
[ 5] 2.00-3.00 sec 72.9 MBytes 612 Mbits/sec 0 1.07 MBytes
[ 5] 3.00-4.00 sec 72.7 MBytes 610 Mbits/sec 0 1.07 MBytes
[ 5] 4.00-5.00 sec 72.7 MBytes 610 Mbits/sec 0 1.07 MBytes
[ 5] 5.00-6.00 sec 72.7 MBytes 610 Mbits/sec 0 1.07 MBytes
[ 5] 6.00-7.00 sec 72.6 MBytes 609 Mbits/sec 0 1.07 MBytes
[ 5] 7.00-8.00 sec 72.6 MBytes 609 Mbits/sec 0 1.07 MBytes
[ 5] 8.00-9.00 sec 72.7 MBytes 609 Mbits/sec 0 1.07 MBytes
[ 5] 9.00-10.00 sec 72.7 MBytes 610 Mbits/sec 0 1.07 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 727 MBytes 610 Mbits/sec 0 sender
[ 5] 0.00-10.57 sec 727 MBytes 577 Mbits/sec receiver
iperf Done.
# iperf3 -R -c 192.168.1.120
Connecting to host 192.168.1.120, port 5201
Reverse mode, remote host 192.168.1.120 is sending
[ 5] local 192.168.1.109 port 10711 connected to 192.168.1.120 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 111 MBytes 932 Mbits/sec
[ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec
[ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec
[ 5] 3.00-4.00 sec 111 MBytes 933 Mbits/sec
[ 5] 4.00-5.00 sec 111 MBytes 933 Mbits/sec
[ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec
[ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec
[ 5] 8.00-9.00 sec 111 MBytes 932 Mbits/sec
[ 5] 9.00-10.00 sec 111 MBytes 933 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.56 sec 1.09 GBytes 885 Mbits/sec 932 sender
[ 5] 0.00-10.00 sec 1.09 GBytes 933 Mbits/sec receiver
Where the server-side reported:
# iperf3 -s
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.109, port 17887
[ 5] local 192.168.1.120 port 5201 connected to 192.168.1.109 port 40422
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 32.1 MBytes 269 Mbits/sec
[ 5] 1.00-2.00 sec 72.7 MBytes 610 Mbits/sec
[ 5] 2.00-3.00 sec 72.7 MBytes 610 Mbits/sec
[ 5] 3.00-4.00 sec 72.5 MBytes 608 Mbits/sec
[ 5] 4.00-5.00 sec 72.7 MBytes 610 Mbits/sec
[ 5] 5.00-6.00 sec 72.6 MBytes 609 Mbits/sec
[ 5] 6.00-7.00 sec 72.6 MBytes 609 Mbits/sec
[ 5] 7.00-8.00 sec 72.6 MBytes 609 Mbits/sec
[ 5] 8.00-9.00 sec 72.6 MBytes 609 Mbits/sec
[ 5] 9.00-10.00 sec 72.7 MBytes 610 Mbits/sec
[ 5] 10.00-10.57 sec 41.3 MBytes 610 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.57 sec 727 MBytes 577 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.109, port 45095
[ 5] local 192.168.1.120 port 5201 connected to 192.168.1.109 port 10711
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 50.3 MBytes 422 Mbits/sec 36 375 KBytes
[ 5] 1.00-2.00 sec 111 MBytes 933 Mbits/sec 95 1.61 MBytes
[ 5] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 92 265 KBytes
[ 5] 3.00-4.00 sec 111 MBytes 932 Mbits/sec 95 342 KBytes
[ 5] 4.00-5.00 sec 111 MBytes 934 Mbits/sec 95 1.01 MBytes
[ 5] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 93 471 KBytes
[ 5] 6.00-7.00 sec 111 MBytes 933 Mbits/sec 96 1.10 MBytes
[ 5] 7.00-8.00 sec 111 MBytes 933 Mbits/sec 100 656 KBytes
[ 5] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 86 676 KBytes
[ 5] 9.00-10.00 sec 111 MBytes 932 Mbits/sec 91 1.08 MBytes
[ 5] 10.00-10.56 sec 62.6 MBytes 933 Mbits/sec 53 696 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.56 sec 1.09 GBytes 885 Mbits/sec 932 sender
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Side note for a networking problem one can
run into on the Rock64 . . .
I'll note that the kernel for the Rock64 still leaks
mbuf_clusters during network activity, not freeing
most of them:
vm.uma.mbuf_cluster.stats.xdomain: 0
vm.uma.mbuf_cluster.stats.fails: 0
vm.uma.mbuf_cluster.stats.frees: 0
vm.uma.mbuf_cluster.stats.allocs: 3302
vm.uma.mbuf_cluster.stats.current: 3302
. . .
vm.uma.mbuf_cluster.limit.max_items: 84417
vm.uma.mbuf_cluster.limit.items: 3302
Copying an about 4 GiByte tar to the Rock64
uses up a large portion of the 84417
max_items : vm.uma.mbuf_cluster.stats.current
ends up being rather large and
vm.uma.mbuf_cluster.stats.frees ends up being
rather small.
===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)
More information about the freebsd-arm
mailing list