Problem with ipfw nat and packet to local services

[Ian Smith]

On Mon, 19 Jul 2010, Mamontov Roman wrote:
 > > What's the value of sysctl net.inet.ip.fw.one_pass ?  It needs to be 0 
 > > so that packets will re-enter the firewall after NAT processing.
 > 
 > > Otherwise, it might help to
 > 
 > > a) run 'ipfw zero' before any tests .. I'm wondering about all those 
 > > packets hitting rule 65535; were they from before adding rule 65000?
 > 
 > > b) add some count rules before and after nat, to show all packets 
 > > that may be eligible for NAT translation, maybe something like:
 > 
 > > 00020 count log ip from any to any in recv ${ext_if}
 > > 00022 count log ip from any to any out xmit ${ext_if}
 > > 00024 count log ip from any to any out recv ${int_if} xmit ${ext_if}
 > 
 > > 00035 nat ...
 > 
 > > 00040 count log ip from any to any in recv ${ext_if}
 > > 00042 count log ip from any to any out xmit ${ext_if}
 > > 00044 count log ip from any to any out recv ${int_if} xmit ${ext_if}
 > 
 > > So you actually get to see the flow of packets before and after nat, 
 > > both to/from the local box and packets mapped to/rom inside addresses.
 > > Again, an 'ipfw zero' before tests will make packet counts clearer.
 > 
 > > Of course something like '# tcpdump -pn -i ext_if' will also show all 
 > > packets via ext_if in some detail.  Be more specific if just looking for 
 > > some particular flows, like maybe appending 'udp port NNNNN' to that.
 > 
 > > That is, try to follow packets you'd expect to be coming in for services 
 > > on the local box so if they are disappearing, you'll know where or why.  
 > > 'netstat -finet -an' will show all those services that are listening.
 > 
 > > If that doesn't help, we'll need more information.
 > 
 > > cheers, Ian
 > 
 > # sysctl net.inet.ip.fw.one_pass
 > net.inet.ip.fw.one_pass: 0
 > 
 > # ipfw show 20-49
 > 00020    40    2016 count log ip from any to me dst-port 22 in recv ext_if1
 > 00021     0       0 count log ip from me 22 to any out xmit ext_if1
 > 00035 13192 9028716 nat 1 ip from any to any via ext_if1
 > 00040     0       0 count log ip from any to me dst-port 22 in recv ext_if1
 > 00041     0       0 count log ip from me 22 to any out xmit ext_if1

Ouch.  It does look like nat is just swallowing the ssh packets on the 
inbound pass .. these surely should appear again after NAT (unchanged).

 > # ipfw nat show config
 > ipfw nat 1 config ip xxx.xxx.xxx.xxx
 > 
 > # tcpdump -pn -i ext_if1 'host yyy.yyy.yyy.yyy'
 > 14:12:48.885011 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS[|tcp]>
 > 14:12:51.888823 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS[|tcp]>
 > 14:12:54.884966 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS[|tcp]>
 > 14:12:57.884090 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460], length 0
 > 14:13:00.885131 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460], length 0
 > 14:13:03.887094 IP yyy.yyy.yyy.yyy.2777 > xxx.xxx.xxx.xxx.22: Flags [S], seq 2880611174, win 65535, options [mss 1460], length 0

Ah, if that 'TS' means TSO - tcpdump(1) doesn't mention either - then 
from the last section of ipfw(8):

     Due to the architecture of libalias(3), ipfw nat is not compatible with
     the TCP segmentation offloading (TSO).  Thus, to reliably nat your net-
     work traffic, please disable TSO on your NICs using ifconfig(8).

ifconfig ext_if1 ?

 > Output
 > # netstat -finet -an | grep yyy.yyy.yyy.yyy
 > is blank.
 >
 > Without rule 35 nat 1 ip from any to any via ext_if1 inbound packet to ssh (for example)
 > pass correctly.
 > 
 > # ipfw delete 35
 > tcpdump -pn -i ext_if 'host yyy.yyy.yyy.yyy'
 > 14:21:45.467233 IP yyy.yyy.yyy.yyy.2790 > xxx.xxx.xxx.xxx.22: Flags [S], seq 376101413, win 65535, options [mss 1460,nop,wscale 1,nop,nop,TS[|tcp]>
 > 14:21:45.467670 IP xxx.xxx.xxx.xxx.22 > xxx.xxx.xxx.xxx.2790: Flags [S.], seq 3270699616, ack 376101414, win 65535, options [mss 1460,nop,wscale 3,nop,nop,TS[|tcp]>

I guess that dest is just mis-obscured, ie yyy.yyy.yyy.yyy:2790

 > 14:21:45.468960 IP yyy.yyy.yyy.yyy.2790 > xxx.xxx.xxx.xxx.22: Flags [.], ack 1, win 33304, options [nop,nop,TS val 40088404 ecr 1166915706], length 0
 > 14:21:45.527438 IP xxx.xxx.xxx.xxx.22 > yyy.yyy.yyy.yyy.2790: Flags [P.], ack 1, win 8326, options [nop,nop,TS val 1166915766 ecr 40088404], length 40
 > 
 > # netstat -finet -an | grep yyy.yyy.yyy.yyy
 > tcp4       0      0 xxx.xxx.xxx.xxx.22        yyy.yyy.yyy.yyy.2790    FIN_WAIT_2
 > 
 > 00020  8 1403 count log ip from any to me dst-port 22 in recv ext_if1
 > 00021  6 2280 count log ip from me 22 to any out xmit ext_if1
 > 00040  8 1403 count log ip from any to me dst-port 22 in recv ext_if1
 > 00041  6 2280 count log ip from me 22 to any out xmit ext_if1
 > 
 > Any ideas?

[later: if you had TSO enabled on the interface, ignore the below ..]

I see you're not logging nat packets now, and wonder if there'd be any 
more clues in the log, but then, does nat log packets it doesn't touch?

Unless I'm completely missing something (wouldn't be the first time :) 
this looks like a bug, and unless someone else has a better clue, you 
should file a PR with what you've demonstrated to date.

cheers, Ian (only vaguely useful with usage, and not at all with code)