Problem with checksum offloading on RPi3 (PF + Jails involved)

Tue Nov 17 04:38:44 UTC 2020

Am 17.11.2020 um 04:04 schrieb YongHyeon PYUN:
> On Mon, Nov 16, 2020 at 12:54:25PM +0100, Carsten Bäcker wrote:
>
> [...]
>
>> i just did a test with your modification.
>> The good news is that the driver itself still works.
> Your test indicates my approach to improve RX validation works.
> Thanks a lot for testing.
>
>> The bad news is that it doesn't solve the problem. :-(
>>
> Let's dig further.  Initially I thought pf(4) could be confused by
> mismatch of received packet length and IP datagram length.  H/W
> does not strip padding bytes for short frame(< 60 bytes) and adds
> FCS bytes such that the minimum ethernet frame length is 64 bytes.
> When RX checksum offloading is enabled, the size will be 66 bytes.
> My patch tried to correct wrong packet length stored in mbuf.
>
> I think simple nping test would be helpful to narrow down the issue
> here. nping is bundled with nmap(ports/security/nmap).
>
> o Test patched smsc(4) with/without pf(4).
>   - Disable RX checksum offloading and make sure all works as
>     expected.
>   - Run the following command on other system and see whether you
>     can get ICMP port unreachable message from a box with smsc(4).
>     5555 is port number that listens on incoming UDP datagrams.
>      #nping -c 1 --udp -p 5555 IPv4_addr_of_remote_host_with_smsc(4)
>
>      #nping -c 1 --udp -p 5555 --data-length 100 IPv4_addr_of_remote_host_with_smsc(4)
>
>   - Repeat the same test with RX checksum offloading enabled.
>
> If there is RX checksum mismatch you wouldn't get ICMP port
> unreachable message.  You can even generate wrong checksummed UDP
> datagrams with --badsum option like the following.
> #nping -c 1 --udp -p 5555 --badsum IPv4_addr_of_remote_host_with_smsc(4)
>
>> I used the snapshot from 2020-10-29 as i already had an sdcard with that
>> image.
>> http://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/13.0/
>>
>> Kernel was build from head + your patch.
>> FreeBSD generic 13.0-CURRENT FreeBSD 13.0-CURRENT #1 r367714M: Mon Nov
>> 16 08:03:24 UTC 2020
>> root at sysbuild:/usr/obj/usr/src_head/arm64.aarch64/sys/GENERIC arm64
>>
>> Log from smsc is attached.
>>
> The log shows my assumption on RX packet layout is correct.
> Thanks.
>
>> As the lack of hardware seems to be a problem... i may be able to
>> provide temporary access, incl. serial using a 2nd pi - but this will
>> take some time as i need to change the network-setup.
>>
> I think it's better to share nping test result first.

Hello,

i attached another set of log-files containing the output of the test
using nping.
The ICMP port unreachable message appeared in each case, but the paket
loss changed to 100%.

In addition i repeated the test with RXCSUM enabled after switching back
to the unpatched kernel.
Not sure if this helps.

Regards,
Carsten

-------------- next part --------------
root at generic:~ # pfctl -sR ; echo ; ifconfig ue0
Disabled

ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80008<VLAN_MTU,LINKSTATE>
        ether b8:27:eb:55:7e:70
        inet 192.168.178.3 netmask 0xffffff00 broadcast 192.168.178.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-17 05:11 CET
SENT (0.0050s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=54261 iplen=28
RCVD (0.0057s) ICMP [192.168.178.3 > 192.168.178.29 Port unreachable (type=3/code=3) ] IP [ttl=64 id=26615 iplen=56 ]

Max rtt: 0.662ms | Min rtt: 0.662ms | Avg rtt: 0.662ms
Raw packets sent: 1 (28B) | Rcvd: 1 (56B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.03 seconds

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 --data-length 100 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-17 05:14 CET
SENT (0.0051s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=5940 iplen=128
RCVD (0.0134s) ICMP [192.168.178.3 > 192.168.178.29 Port 5555 unreachable (type=3/code=3) ] IP [ttl=64 id=55538 iplen=156 ]

Max rtt: 8.268ms | Min rtt: 8.268ms | Avg rtt: 8.268ms
Raw packets sent: 1 (128B) | Rcvd: 1 (156B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.05 seconds

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 --badsum 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-17 05:15 CET
SENT (0.0050s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=40572 iplen=28

Max rtt: N/A | Min rtt: N/A | Avg rtt: N/A
Raw packets sent: 1 (28B) | Rcvd: 0 (0B) | Lost: 1 (100.00%)
Nping done: 1 IP address pinged in 1.04 seconds
-------------- next part --------------
root at generic:~ # pfctl -sR ; echo ; ifconfig ue0
Disabled

ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
        ether b8:27:eb:55:7e:70
        inet 192.168.178.3 netmask 0xffffff00 broadcast 192.168.178.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-17 05:18 CET
SENT (0.0050s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=36202 iplen=28
RCVD (0.0058s) ICMP [192.168.178.3 > 192.168.178.29 Port unreachable (type=3/code=3) ] IP [ttl=64 id=55539 iplen=56 ]

Max rtt: 0.672ms | Min rtt: 0.672ms | Avg rtt: 0.672ms
Raw packets sent: 1 (28B) | Rcvd: 1 (56B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.03 seconds

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 --data-length 100 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-17 05:18 CET
SENT (0.0050s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=59079 iplen=128
RCVD (0.0059s) ICMP [192.168.178.3 > 192.168.178.29 Port 5555 unreachable (type=3/code=3) ] IP [ttl=64 id=55540 iplen=156 ]

Max rtt: 0.799ms | Min rtt: 0.799ms | Avg rtt: 0.799ms
Raw packets sent: 1 (128B) | Rcvd: 1 (156B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.06 seconds

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 --badsum 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-17 05:18 CET
SENT (0.0050s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=9384 iplen=28

Max rtt: N/A | Min rtt: N/A | Avg rtt: N/A
Raw packets sent: 1 (28B) | Rcvd: 0 (0B) | Lost: 1 (100.00%)
Nping done: 1 IP address pinged in 1.07 seconds
-------------- next part --------------
root at generic:~ # uname -a
FreeBSD generic 13.0-CURRENT FreeBSD 13.0-CURRENT #0 b9403d7aae8-c254071(main): Thu Oct 29 10:38:29 UTC 2020     root at releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC  arm64
root at generic:~ # pfctl -sR ; echo ; ifconfig ue0
Disabled

ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
        ether b8:27:eb:55:7e:70
        inet 192.168.178.3 netmask 0xffffff00 broadcast 192.168.178.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-17 05:24 CET
SENT (0.0050s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=27202 iplen=28
RCVD (0.0063s) ICMP [192.168.178.3 > 192.168.178.29 Port unreachable (type=3/code=3) ] IP [ttl=64 id=63658 iplen=56 ]

Max rtt: 1.243ms | Min rtt: 1.243ms | Avg rtt: 1.243ms
Raw packets sent: 1 (28B) | Rcvd: 1 (56B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.02 seconds

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 --data-length 100 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-17 05:24 CET
SENT (0.0050s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=64786 iplen=128
RCVD (0.0058s) ICMP [192.168.178.3 > 192.168.178.29 Port 5555 unreachable (type=3/code=3) ] IP [ttl=64 id=56542 iplen=156 ]

Max rtt: 0.741ms | Min rtt: 0.741ms | Avg rtt: 0.741ms
Raw packets sent: 1 (128B) | Rcvd: 1 (156B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.05 seconds

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 --badsum 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-17 05:24 CET
SENT (0.0050s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=32046 iplen=28

Max rtt: N/A | Min rtt: N/A | Avg rtt: N/A
Raw packets sent: 1 (28B) | Rcvd: 0 (0B) | Lost: 1 (100.00%)
Nping done: 1 IP address pinged in 1.07 seconds