Problem with checksum offloading on RPi3 (PF + Jails involved)

Wed Nov 18 09:47:12 UTC 2020

Am 18.11.2020 um 05:48 schrieb YongHyeon PYUN:
> On Tue, Nov 17, 2020 at 05:38:38AM +0100, Carsten Bäcker wrote:
>
> [...]
>
>> i attached another set of log-files containing the output of the test
>> using nping.
>> The ICMP port unreachable message appeared in each case, but the paket
>> loss changed to 100%.
> That came from 3rd nping test with --badsum option and that's normal.
> If I read attached log-files correctly all worked as expected.
> I don't know what pf rulesets you used. Did you use the same pf
> ruleset + jail with patched smsc(4)?  If you didn't yet could you
> test it again and let me know nping output?

Sorry, that's my fault - i overlooked your request for a test with pf
enabled.
The example-ruleset is attached again. I added a line to allow the
incoming ping to 5555.

I don't see a difference until i enable the redirection to the jail
which makes the packet with extended data-length fail.
Once i disable RXCSUM it works.

Regards,
Carsten

-------------- next part --------------
ext_if = "ue0"

wwwpublicjail = 10.0.0.2

set block-policy return
set skip on lo0
set skip on lo1

table <jails> persist

nat on $ext_if from { $wwwpublicjail } to any -> ($ext_if)
#rdr on $ext_if inet proto udp from any to ($ext_if) port 5555 -> $wwwpublicjail

block in all

pass in inet proto icmp all icmp-type echoreq
pass in inet proto tcp to port { ssh }
pass in inet proto udp to port { 5555 }

pass out all

-------------- next part --------------
root at generic:~ # pfctl -sr ; echo ; pfctl -sn ; echo ; ifconfig ue0 ; echo ; uname -v
block return in all
pass in inet proto icmp all icmp-type echoreq keep state
pass in inet proto tcp from any to any port = ssh flags S/SA keep state
pass in inet proto udp from any to any port = rplay keep state
pass out all flags S/SA keep state

nat on ue0 inet from 10.0.0.2 to any -> (ue0) round-robin

ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
        ether b8:27:eb:55:7e:70
        inet 192.168.178.3 netmask 0xffffff00 broadcast 192.168.178.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

FreeBSD 13.0-CURRENT #1 r367714M: Mon Nov 16 08:03:24 UTC 2020     root at sysbuild:/usr/obj/usr/src_head/arm64.aarch64/sys/GENERIC

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-18 09:30 CET
SENT (0.0048s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=12801 iplen=28
RCVD (0.0057s) ICMP [192.168.178.3 > 192.168.178.29 Port unreachable (type=3/code=3) ] IP [ttl=64 id=36001 iplen=56 ]

Max rtt: 0.823ms | Min rtt: 0.823ms | Avg rtt: 0.823ms
Raw packets sent: 1 (28B) | Rcvd: 1 (56B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.05 seconds

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 --data-length 100 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-18 09:32 CET
SENT (0.0048s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=40512 iplen=128
RCVD (0.0058s) ICMP [192.168.178.3 > 192.168.178.29 Port 5555 unreachable (type=3/code=3) ] IP [ttl=64 id=56235 iplen=156 ]

Max rtt: 0.839ms | Min rtt: 0.839ms | Avg rtt: 0.839ms
Raw packets sent: 1 (128B) | Rcvd: 1 (156B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.03 seconds
-------------- next part --------------
root at generic:~ # pfctl -sr ; echo ; pfctl -sn ; echo ; ifconfig ue0 ; echo ; uname -v
block return in all
pass in inet proto icmp all icmp-type echoreq keep state
pass in inet proto tcp from any to any port = ssh flags S/SA keep state
pass in inet proto udp from any to any port = rplay keep state
pass out all flags S/SA keep state

nat on ue0 inet from 10.0.0.2 to any -> (ue0) round-robin
rdr on ue0 inet proto udp from any to (ue0) port = rplay -> 10.0.0.2

ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80009<RXCSUM,VLAN_MTU,LINKSTATE>
        ether b8:27:eb:55:7e:70
        inet 192.168.178.3 netmask 0xffffff00 broadcast 192.168.178.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

FreeBSD 13.0-CURRENT #1 r367714M: Mon Nov 16 08:03:24 UTC 2020     root at sysbuild:/usr/obj/usr/src_head/arm64.aarch64/sys/GENERIC

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-18 09:34 CET
SENT (0.0048s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=853 iplen=28
RCVD (0.0058s) ICMP [192.168.178.3 > 192.168.178.29 Port unreachable (type=3/code=3) ] IP [ttl=64 id=56236 iplen=56 ]

Max rtt: 0.814ms | Min rtt: 0.814ms | Avg rtt: 0.814ms
Raw packets sent: 1 (28B) | Rcvd: 1 (56B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.04 seconds

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 --data-length 100 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-18 09:35 CET
SENT (0.0047s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=21585 iplen=128

Max rtt: N/A | Min rtt: N/A | Avg rtt: N/A
Raw packets sent: 1 (128B) | Rcvd: 0 (0B) | Lost: 1 (100.00%)
Nping done: 1 IP address pinged in 1.01 seconds
-------------- next part --------------
root at generic:~ # pfctl -sr ; echo ; pfctl -sn ; echo ; ifconfig ue0 ; echo ; uname -v
block return in all
pass in inet proto icmp all icmp-type echoreq keep state
pass in inet proto tcp from any to any port = ssh flags S/SA keep state
pass in inet proto udp from any to any port = rplay keep state
pass out all flags S/SA keep state

nat on ue0 inet from 10.0.0.2 to any -> (ue0) round-robin
rdr on ue0 inet proto udp from any to (ue0) port = rplay -> 10.0.0.2

ue0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80008<VLAN_MTU,LINKSTATE>
        ether b8:27:eb:55:7e:70
        inet 192.168.178.3 netmask 0xffffff00 broadcast 192.168.178.255
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

FreeBSD 13.0-CURRENT #1 r367714M: Mon Nov 16 08:03:24 UTC 2020     root at sysbuild:/usr/obj/usr/src_head/arm64.aarch64/sys/GENERIC

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-18 10:27 CET
SENT (0.0048s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=54705 iplen=28
RCVD (0.0057s) ICMP [192.168.178.3 > 192.168.178.29 Port unreachable (type=3/code=3) ] IP [ttl=64 id=16013 iplen=56 ]

Max rtt: 0.788ms | Min rtt: 0.788ms | Avg rtt: 0.788ms
Raw packets sent: 1 (28B) | Rcvd: 1 (56B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.04 seconds

---
root at test-amd64:~ # nping -c 1 --udp -p 5555 --data-length 100 192.168.178.3

Starting Nping 0.7.80 ( https://nmap.org/nping ) at 2020-11-18 10:28 CET
SENT (0.0049s) UDP 192.168.178.29:53 > 192.168.178.3:5555 ttl=64 id=61299 iplen=128
RCVD (0.0059s) ICMP [192.168.178.3 > 192.168.178.29 Port 5555 unreachable (type=3/code=3) ] IP [ttl=64 id=25452 iplen=156 ]

Max rtt: 0.919ms | Min rtt: 0.919ms | Avg rtt: 0.919ms
Raw packets sent: 1 (128B) | Rcvd: 1 (156B) | Lost: 0 (0.00%)
Nping done: 1 IP address pinged in 1.07 seconds