IPFW fwd not working after upgrade from 9.2 to 10.0

[John Nielsen]

I have been using IPFW FWD to do per-interface routing on a VM instance. The default gateway is on interface vtnet0, but there is a second interface, vtnet1, on a different network with its own public IP address. The second network has its own gateway, which I'd like to use for responses to connections coming on on vtnet1. Under 9.2, the below worked fine:

fwd ${GW2} ip from ${PUBIP2} to not table(120) out via vtnet0

Table 120 contains all the local networks for which I don't want the rule to apply.

I updated the VM to 10.0-RELEASE, with no changes to the IPFW rules or network configuration. The forwarding to the secondary router no longer works. Traffic comes in on ${PUBIP2} fine, and the counter for the IPFW rule increments, but no packets are actually sent out vtnet1. Instead, it's trying to do a weird ARP query:


# tcpdump -n -p -i vtnet1
...
16:46:33.146324 IP ${OUTSIDE_IP}.55063 > ${PUBIP2}.22: Flags [S], seq 2242981455, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val 1978614336 ecr 0,sackOK,eol], length 0
16:46:33.146372 ARP, Request who-has ${GW1} tell ${PUBIP2}, length 28

If I try to SSH from an outside IP to the public IP on vtnet1, a response never goes out either interface (vtnet0 or vtnet1). Instead, an ARP query is going out (on vtnet1) looking for the default gateway IP, which is only reachable on vtnet0.

On the off chance this is not a bug, is there a better way I should be doing per-interface routing under FreeBSD 10? If it is a bug, can anyone suggest what might be going on here and how to track it down further?

Thanks,

JN