IPFW NAT behaviour different on 10-Stable versus 11-Stable

Sat Sep 2 01:45:00 UTC 2017

I have a problem that seems to be a difference between ipfw/NAT 
behaviour in 10-Stable versus 11-Stable. I have two servers: one running 
10-Stable and one running 11-Stable. I'm using the same rule set on both 
(see below). It works correctly on 10-Stable but not on 11.

The problem is seen on two places: an outgoing SMTP connection on port 
465, and an incoming to an IMAP server on port 993. In both cases, there 
are lost packets and retransmissions. See below for a tshark capture of 
one attempted SMTP session.

Setting sysctl net.inet.ip.fw.one_pass to one or zero makes no 
difference. Deleting the sshguard rule (table 22) makes no difference. 
Deleting the nat rule makes everything work for this SMTP session (but 
breaks the other machines on my network obviously).

I have no doubt that I have misconfigured the firewall, but I don't see 
what. And why is 11 different to 10? Any help would be much appreciated.

Thanks in advance,

     Graham

Tshark:

(XXX is the SMTP server, YYY is my public IP address)

root# tshark -Y tcp.port==465 -i igb1
Capturing on 'igb1'
     4   0.722919 YYY → XXX TLSv1.2 180 Client Key Exchange, Change 
Cipher Spec, Encrypted Handshake Message
   527  17.822843 YYY → XXX TCP 180 [TCP Retransmission] 63024 → 465 
[PSH, ACK] Seq=1 Ack=1 Win=65535 Len=126
  1335  51.814540 YYY → XXX TCP 180 [TCP Retransmission] 63024 → 465 
[PSH, ACK] Seq=1 Ack=1 Win=65535 Len=126
  1393  85.806537 YYY → XXX TCP 180 [TCP Retransmission] 63024 → 465 
[PSH, ACK] Seq=1 Ack=1 Win=65535 Len=126
  2142 107.799346 XXX → YYY TCP 60 465 → 63024 [FIN, ACK] Seq=1 Ack=1 
Win=15544 Len=0
  2143 107.799393 YYY → XXX TCP 54 63024 → 465 [ACK] Seq=127 Ack=2 
Win=65535 Len=0
  2144 107.800135 YYY → XXX TCP 54 63024 → 465 [FIN, ACK] Seq=127 Ack=2 
Win=65535 Len=0
  2145 107.822047 YYY → XXX TCP 74 53762 → 465 [SYN] Seq=0 Win=65535 
Len=0 MSS=1460 WS=64 SACK_PERM=1 TSval=2591962 TSecr=0
  2146 107.977234 XXX → YYY TCP 60 465 → 63024 [RST] Seq=2 Win=0 Len=0
  2149 108.001214 XXX → YYY TCP 62 465 → 53762 [SYN, ACK] Seq=0 Ack=1 
Win=14600 Len=0 MSS=1460 SACK_PERM=1
  2150 108.001270 YYY → XXX TCP 54 53762 → 465 [ACK] Seq=1 Ack=1 
Win=65535 Len=0
  2151 108.009014 YYY → XXX TLSv1 323 Client Hello
  2160 108.187708 XXX → YYY TCP 60 465 → 53762 [ACK] Seq=1 Ack=270 
Win=15544 Len=0
  2176 108.687644 XXX → YYY TLSv1.2 1514 Server Hello
  2177 108.687884 XXX → YYY TCP 1514 465 → 53762 [PSH, ACK] Seq=1461 
Ack=270 Win=15544 Len=1460 [TCP segment of a reassembled PDU]
  2178 108.687949 YYY → XXX TCP 54 53762 → 465 [ACK] Seq=270 Ack=2921 
Win=62874 Len=0
  2179 108.688175 XXX → YYY TCP 1230 465 → 53762 [PSH, ACK] Seq=2921 
Ack=270 Win=15544 Len=1176 [TCP segment of a reassembled PDU]
  2180 108.704012 XXX → YYY TCP 1514 465 → 53762 [ACK] Seq=4097 Ack=270 
Win=15544 Len=1460 [TCP segment of a reassembled PDU]
  2181 108.704052 YYY → XXX TCP 54 53762 → 465 [ACK] Seq=270 Ack=5557 
Win=64240 Len=0
  2182 108.704625 XXX → YYY TLSv1.2 969 Certificate, Server Key 
Exchange, Server Hello Done
  2183 108.715222 YYY → XXX TLSv1.2 180 Client Key Exchange, Change 
Cipher Spec, Encrypted Handshake Message
  2211 109.133829 YYY → XXX TCP 180 [TCP Retransmission] 53762 → 465 
[PSH, ACK] Seq=270 Ack=6472 Win=65535 Len=126
  2238 109.443030 XXX → YYY TCP 969 [TCP Spurious Retransmission] 465 → 
53762 [PSH, ACK] Seq=5557 Ack=270 Win=15544 Len=915[Reassembly error, 
protocol TCP: New fragment overlaps old data (retransmission?)]
  2239 109.443099 YYY → XXX TCP 54 [TCP Dup ACK 2183#1] 53762 → 465 
[ACK] Seq=396 Ack=6472 Win=65535 Len=0
  2244 109.772021 YYY → XXX TCP 180 [TCP Retransmission] 53762 → 465 
[PSH, ACK] Seq=270 Ack=6472 Win=65535 Len=126
  2301 110.827331 YYY → XXX TCP 180 [TCP Retransmission] 53762 → 465 
[PSH, ACK] Seq=270 Ack=6472 Win=65535 Len=126
  2402 112.770796 YYY → XXX TCP 180 [TCP Retransmission] 53762 → 465 
[PSH, ACK] Seq=270 Ack=6472 Win=65535 Len=126
  2612 116.391551 YYY → XXX TCP 180 [TCP Retransmission] 53762 → 465 
[PSH, ACK] Seq=270 Ack=6472 Win=65535 Len=126
  2711 119.018591 YYY → XXX TCP 180 [TCP Retransmission] 53762 → 465 
[PSH, ACK] Seq=270 Ack=6472 Win=65535 Len=126
  2737 123.957850 YYY → XXX TCP 180 [TCP Retransmission] 53762 → 465 
[PSH, ACK] Seq=270 Ack=6472 Win=65535 Len=126
  2789 133.632511 YYY → XXX TCP 180 [TCP Retransmission] 53762 → 465 
[PSH, ACK] Seq=270 Ack=6472 Win=65535 Len=126
  2859 152.776509 YYY → XXX TCP 180 [TCP Retransmission] 53762 → 465 
[PSH, ACK] Seq=270 Ack=6472 Win=65535 Len=126
^C32 packets captured
root#

Rules:

# stop spoofing
add deny all from LAN_NET to any in via OUTSIDE_IF
add deny all from WIFI_NET to any in via OUTSIDE_IF

# allow anything on the LAN
add allow all from any to any via LAN_IF

# and from the VPN
add allow all from any to any via VPN_IF

# allow anything from the wireless network to the outside world (but not 
to the LAN)
add allow ip from any to not LAN_NET via WIFI_IF

# create a table of addresses to block
#table 1 destroy
#table 1 create type addr
table 1 flush
# add RFC1918 nets
table 1 add 10.0.0.0/8
table 1 add 172.16.0.0/12
table 1 add 192.168.0.0/16
# and draft-manning-dsua-03.txt nets
table 1 add 0.0.0.0/8
table 1 add 169.254.0.0/16
table 1 add 192.0.2.0/24
table 1 add 224.0.0.0/4
table 1 add 240.0.0.0/4
# stop entries in the table coming in on the outside interface
add deny all from table(1) to any in recv OUTSIDE_IF

# similarly for IPv6
#table 2 destroy
#table 2 create type addr
table 2 flush
# Stop unique local unicast address on the outside interface
table 2 add fc00::/7
# Stop site-local on the outside interface
table 2 add fec0::/10
# Disallow "internal" addresses to appear on the wire.
table 2 add ::ffff:0.0.0.0/96
# Disallow packets to malicious IPv4 compatible prefix.
#table 2 add ::224.0.0.0/100 gives error "Use IPv4 instead of v4-compatible"
#table 2 add ::127.0.0.0/104 ditto
table 2 add ::0.0.0.0/104
#table 2 add ::255.0.0.0/104 ditto
#
table 2 add ::0.0.0.0/96
# Disallow packets to malicious 6to4 prefix.
table 2 add 2002:e000::/20
table 2 add 2002:7f00::/24
table 2 add 2002:0000::/24
table 2 add 2002:ff00::/24
#
table 2 add 2002:0a00::/24
table 2 add 2002:ac10::/28
table 2 add 2002:c0a8::/32
#
table 2 add ff05::/16
# block these addresses both incoming and outgoing
add deny all from table(2) to any via IPV6_IF
add deny all from any to table(2) via IPV6_IF

# block sshguard entries
add reset ip from table(22) to me

# allow setup of incoming SSH, IMAPS, and OpenVPN
add allow tcp from any to me ssh setup
add allow tcp from any to me6 ssh setup
add allow tcp from any to me imaps setup
add allow tcp from any to me6 imaps setup
add allow tcp from any to me openvpn setup
add allow tcp from any to me6 openvpn setup
add allow udp from any to me openvpn

# allow IPP, IMAPS, and SMTP from wireless
add allow ip from any to LAN_NET dst-port printer setup via WIFI_IF
add allow ip from any to me dst-port ipp setup via WIFI_IF
add allow ip from any to me dst-port smtp setup via WIFI_IF
add allow ip from any to me dst-port imaps setup via WIFI_IF

# allow some ICMP types but nothing else
add allow icmp from any to any icmptypes 0,3,8,11
add deny icmp from any to any

#add allow ipv6 from any to any

# NAT
# redirect ports to PS4
nat 1 config if OUTSIDE_IF same_ports deny_in redirect_port tcp 
PS4_ADDR:1935 1935 redirect_port tcp PS4_ADDR:3478 3478 redirect_port 
tcp PS4_ADDR:3479 3479 redirect_port tcp PS4_ADDR:3480 3480 
redirect_port udp PS4_ADDR:3478 3478 redirect_port udp PS4_ADDR:3479 3479
add nat 1 ip4 from any to any via OUTSIDE_IF

# and block the above table again outbound
add deny all from table(1) to any out xmit OUTSIDE_IF

# allow TCP through if setup succeeded
add pass tcp from any to any established

# allow IP fragments to pass through
add pass all from any to any frag

# allow TCP ports needed for PS4
add allow tcp from any to PS4_ADDR 1935 in via OUTSIDE_IF setup
add allow tcp from any to PS4_ADDR 3478 in via OUTSIDE_IF setup
add allow tcp from any to PS4_ADDR 3479 in via OUTSIDE_IF setup
add allow tcp from any to PS4_ADDR 3480 in via OUTSIDE_IF setup
add allow udp from any to PS4_ADDR 3478 in via OUTSIDE_IF
add allow udp from any to PS4_ADDR 3479 in via OUTSIDE_IF

# allow DNS & NTP queries out to the world (and their replies back in)
add allow udp from me to any 53 keep-state
add allow udp from me to any 123 keep-state
# but no other UDP in from outside
add deny udp from any to any in via OUTSIDE_IF
# and allow any other UDP
add allow udp from any to any

# reject all setup of incoming connections from the outside
add deny tcp from any to any in via OUTSIDE_IF setup

# reject all setup of incoming connections from the IPV6 tunnel
add deny tcp from any to any in via gif0 setup

# reject all setup of incoming connections from the wireless
add deny tcp from any to any in via WIFI_IF setup

# allow setup of any other TCP connection
add pass tcp from any to any setup

# Everything else is denied by default, unless the 
IPFIREWALL_DEFAULT_TO_ACCEPT option is set in your kernel config file. 
But we add this rule anyway to allow logging.
add deny all from any to any