IPFW fwd not working after upgrade from 9.2 to 10.0

[Ronald Klop]

On Thu, 06 Feb 2014 01:08:24 +0100, John Nielsen <lists at jnielsen.net>  
wrote:

> I have been using IPFW FWD to do per-interface routing on a VM instance.  
> The default gateway is on interface vtnet0, but there is a second  
> interface, vtnet1, on a different network with its own public IP  
> address. The second network has its own gateway, which I'd like to use  
> for responses to connections coming on on vtnet1. Under 9.2, the below  
> worked fine:
>
> fwd ${GW2} ip from ${PUBIP2} to not table(120) out via vtnet0
>
> Table 120 contains all the local networks for which I don't want the  
> rule to apply.
>
> I updated the VM to 10.0-RELEASE, with no changes to the IPFW rules or  
> network configuration. The forwarding to the secondary router no longer  
> works. Traffic comes in on ${PUBIP2} fine, and the counter for the IPFW  
> rule increments, but no packets are actually sent out vtnet1. Instead,  
> it's trying to do a weird ARP query:
>
>
> # tcpdump -n -p -i vtnet1
> ...
> 16:46:33.146324 IP ${OUTSIDE_IP}.55063 > ${PUBIP2}.22: Flags [S], seq  
> 2242981455, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val  
> 1978614336 ecr 0,sackOK,eol], length 0
> 16:46:33.146372 ARP, Request who-has ${GW1} tell ${PUBIP2}, length 28
>
> If I try to SSH from an outside IP to the public IP on vtnet1, a  
> response never goes out either interface (vtnet0 or vtnet1). Instead, an  
> ARP query is going out (on vtnet1) looking for the default gateway IP,  
> which is only reachable on vtnet0.
>
> On the off chance this is not a bug, is there a better way I should be  
> doing per-interface routing under FreeBSD 10? If it is a bug, can anyone  
> suggest what might be going on here and how to track it down further?
>
> Thanks,
>
> JN

The errata of FreeBSD 10.0 mentions ipfw fwd.
http://www.freebsd.org/releases/10.0R/errata.html

Ronald.