ipsec with ipfw divert (not NAT) encodes a packet twice breaking PMTUD

[Julian Elischer]

Eugene Grosbein wrote:

>>Submitter-Id:	current-users
>>Originator:	Eugene Grosbein
>>Organization:	Svyaz Service JSC
>>Confidential:	no
>>Synopsis:	ipsec with ipfw divert (not NAT) encodes a packet twice breaking PMTUD
>>Severity:	serious
>>Priority:	high
>>Category:	kern
>>Class:		sw-bug
>>Release:	FreeBSD 6.1-STABLE i386
>>Environment:
>>    
>>
>System: FreeBSD nkz.delikates-nk.ru 6.1-STABLE FreeBSD 6.1-STABLE #1: Thu Sep 7 13:31:53 KRAST 2006 root at nkz.delikates-nk.ru:/home/obj/home/src/sys/NKZ i386
>	options IPDIVERT
>	options IPSEC
>	options IPSEC_ESP
>
>  
>
>>Description:
>>    
>>
>	When outgoing packet encoded due to corresponding IPSEC policy
>	is passed to divert socket (f.e. to ipacctd for accounting),
>	it is encoded second time with IPSEC then. Besides obvious
>	logic error, this also results in broken Path MTU Discovery.
>  
>

unfortunatly this comes from the fact that divert returns packets to the
kernel by passing them to the IP stack and letting them be processed 
again. 
There is a flag that is set to allow the ipfw to know that they
have been seen before, and it is possible that one could make IPSEC notice
that flag as well but it would be pretty hacky.  One other solution 
would be
to make some way in which ipdivert can really inject a packet back at the
point where it was extracted but that would probably require spliting 
ip_output()
into two functions, (as was done in ether_output() ) but that would 
probably
not 'fly' very well.

>  
>
>>How-To-Repeat:
>>    
>>
>
>	Use a kernel with options IPDIVERT, IPSEC, IPSEC_ESP
>	(my kernel also contains IPSEC_FILTERGIF, but this should not matter).
>
>	Suppose there are two local nets numbered 192.168.1.0/24
>	and 192.168.2.0/24, each has a FreeBSD router
>	(192.168.1.1 and 192.168.2.1). Routers make gif(4) tunnel between
>	and use IPSEC transport mode to encrypt its contents.
>	Their external IP addresses are 1.1.1.1 and 2.2.2.2
>
>	Here is /etc/ipsec.conf:
>
>add 1.1.1.1  81.16.143.102 esp 1007 -m transport -E blowfish-cbc "xxx";
>add 2.2.2.2 1.1.1.1 esp 2007 -m transport -E blowfish-cbc "yyy";
>
>spdadd 1.1.1.1/32 2.2.2.2/32 any -P in  ipsec esp/transport/1.1.1.1-2.2.2.2/require;
>spdadd 2.2.2.2/32 1.1.1.1/32 any -P out ipsec esp/transport/2.2.2.2-1.1.1.1/require;
>
>	Another router has similar /etc/ipsec.conf.
>
>	Use this script to prepare IPSEC keys for tcpdump:
>
>#!/bin/sh
>
>setkey -D | awk '
>  /^[1-9]/  { ip=$2; }
>  $1=="esp" { 
>		sub(/spi=[^(]+\(/, "", $3);
>		sub(/\)/, "", $3);
>		printf"%s@%s ",$3,ip;
>	    }
>  $1=="E:"  { printf "%s:0x%s%s%s%s\n", $2, $3, $4, $5, $6; }
>' > /tmp/keys.txt
>
>	When we use one of the routers to run
>	"tcpdump -s0 -n -p -i $iface -E /tmp/keys.txt esp",
>	we see that traffic is encoded with ESP and decoded properly,
>	still good.
>
>	Now install ports/net-mgmt/ipacctd, run it manually with
>	"ipacctd -p 4000", then command:
>	"ipfw add 10 divert 4000 ip from any to any out"
>	and rerun tcpdump command shown above. You'll see that
>	outgoing packets are encapsulated with ESP twice.
>	The same you'll see at the other side for incoming packets.
>
>  
>
>>Fix:
>>    
>>
>
>	Unknown. The only known workaround is to avoid diverting ESP.
>	This workaroung is not always acceptable.
>
>
>Eugene Grosbein
>_______________________________________________
>freebsd-net at freebsd.org mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-net
>To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>  
>