kern/103135: ipsec with ipfw divert (not NAT) encodes a packet twice breaking PMTUD

Eugene Grosbein eugen at
Mon Sep 11 06:50:22 PDT 2006

>Number:         103135
>Category:       kern
>Synopsis:       ipsec with ipfw divert (not NAT) encodes a packet twice breaking PMTUD
>Confidential:   no
>Severity:       serious
>Priority:       high
>Responsible:    freebsd-bugs
>State:          open
>Class:          sw-bug
>Submitter-Id:   current-users
>Arrival-Date:   Mon Sep 11 13:50:16 GMT 2006
>Originator:     Eugene Grosbein
>Release:        FreeBSD 6.1-STABLE i386
Svyaz Service JSC
System: FreeBSD 6.1-STABLE FreeBSD 6.1-STABLE #1: Thu Sep 7 13:31:53 KRAST 2006 root at i386
	options IPDIVERT
	options IPSEC
	options IPSEC_ESP

	When outgoing packet encoded due to corresponding IPSEC policy
	is passed to divert socket (f.e. to ipacctd for accounting),
	it is encoded second time with IPSEC then. Besides obvious
	logic error, this also results in broken Path MTU Discovery.


	Use a kernel with options IPDIVERT, IPSEC, IPSEC_ESP
	(my kernel also contains IPSEC_FILTERGIF, but this should not matter).

	Suppose there are two local nets numbered
	and, each has a FreeBSD router
	( and Routers make gif(4) tunnel between
	and use IPSEC transport mode to encrypt its contents.
	Their external IP addresses are and

	Here is /etc/ipsec.conf:

add esp 1007 -m transport -E blowfish-cbc "xxx";
add esp 2007 -m transport -E blowfish-cbc "yyy";

spdadd any -P in  ipsec esp/transport/;
spdadd any -P out ipsec esp/transport/;

	Another router has similar /etc/ipsec.conf.

	Use this script to prepare IPSEC keys for tcpdump:


setkey -D | awk '
  /^[1-9]/  { ip=$2; }
  $1=="esp" { 
		sub(/spi=[^(]+\(/, "", $3);
		sub(/\)/, "", $3);
		printf"%s@%s ",$3,ip;
  $1=="E:"  { printf "%s:0x%s%s%s%s\n", $2, $3, $4, $5, $6; }
' > /tmp/keys.txt

	When we use one of the routers to run
	"tcpdump -s0 -n -p -i $iface -E /tmp/keys.txt esp",
	we see that traffic is encoded with ESP and decoded properly,
	still good.

	Now install ports/net-mgmt/ipacctd, run it manually with
	"ipacctd -p 4000", then command:
	"ipfw add 10 divert 4000 ip from any to any out"
	and rerun tcpdump command shown above. You'll see that
	outgoing packets are encapsulated with ESP twice.
	The same you'll see at the other side for incoming packets.


	Unknown. The only known workaround is to avoid diverting ESP.
	This workaroung is not always acceptable.

Eugene Grosbein

More information about the freebsd-bugs mailing list