MAC address / per-proto ARP caching in 8.1-RELEASE

[Steve Polyack]

Is anyone aware of some sort of facility in either FreeBSD 8.1-RELEASE 
or the em(4) driver which would cause it to cache MAC addresses / ARP 
entries for hosts on a per-protocol basis?  We've been doing some 
testing with new routers, and almost every time we switch them in or out 
our FreeBSD machines get stuck sending UDP DNS requests to the MAC 
address of the old default gateway in complete disregard for the current 
contents of the ARP table.  New TCP connections do not seem to be affected:

[spolyack at web01 ~]$ uname -a
FreeBSD web01 8.1-RELEASE-p2 FreeBSD 8.1-RELEASE-p2 #3: Mon Dec  6 
08:58:21 EST 2010     root at web01:/usr/obj/usr/src/sys/WEB-AMD64  amd64

Current ARP table after the router replacement, the default gateway is 
10.0.1.254, which it has the correct new MAC address for already:
[spolyack at web01 ~]$ arp -an
? (10.0.1.17) at 00:0c:29:47:74:3a on em2 permanent [ethernet]
? (10.0.1.130) at 00:0c:29:47:74:26 on em0 permanent [ethernet]
? (10.0.1.254) at 00:a0:c9:00:01:01 on em0 expires in 915 seconds [ethernet]
? (10.0.0.17) at 00:0c:29:47:74:30 on em1 permanent [ethernet]
? (10.0.0.15) at 00:0c:29:47:74:30 on em1 permanent [ethernet]
? (10.0.0.11) at 00:0c:29:47:74:30 on em1 permanent [ethernet]
? (10.0.0.2) at 00:1f:a0:10:28:70 on em1 expires in 1162 seconds [ethernet]
? (10.0.0.1) at 00:1f:a0:10:28:70 on em1 expires in 943 seconds [ethernet]

tcpdump shows the DNS requests heading to the *old* router's MAC address 
despite the contents of the ARP table:
[spolyack at web01 ~]$ sudo tcpdump -i em0 -s 256 -vvv -e -n -ttt 'port 53'
tcpdump: listening on em0, link-type EN10MB (Ethernet), capture size 256 
bytes
00:00:00.000000 00:0c:29:47:74:26 > 54:75:d0:a3:7c:8c, ethertype IPv4 
(0x0800), length 86: (tos 0x0, ttl 64, id 55590, offset 0, flags [none], 
proto UDP (17), length 72)
     10.0.1.130.52419 > 10.0.2.80.53: [bad udp cksum fdc5!] 52051+ A? 
db-testing-lab. (44)

[spolyack at web01 ~]$ arp -an | grep 54:75:d0
[spolyack at web01 ~]$

Using telnet to form a TCP connection to port 53 on the same DNS server 
which the above UDP requests are headed to works just fine:
[spolyack at web01 ~]$ telnet 10.0.2.80 53
Trying 10.0.2.80...
Connected to 10.0.2.80.
Escape character is '^]'.
00:03:43.272134 00:0c:29:47:74:26 > 00:a0:c9:00:01:01, ethertype IPv4 
(0x0800), length 74: (tos 0x10, ttl 64, id 24383, offset 0, flags [DF], 
proto TCP (6), length 60)
     10.0.2.130.20130 > 10.0.2.80.53: Flags [S], cksum 0x0353 (incorrect 
-> 0xf74d), seq 2674341615, win 65535, options [mss 1460,nop,wscale 
3,sackOK,TS val 60433610 ecr 0], length
...

Even after the above, standard UDP DNS requests are still headed to the 
old default gateway MAC address.  tcpdumping and looking at ARP 
requests/responses doesn't show any traces of the old MAC address 
either.  The switch which connects the server and router doesn't have 
any entries referencing the old router's MAC address anywhere either.  
I'm assuming this isn't related to the new flowtable features included 
in 8.x, since they appear to function on a 4-tuple of src addr, src 
port, dst addr, and dst port, which isn't going to match for new DNS 
requests.

If anyone has any ideas or suggestions on what additional data I can 
grab, I'd be happy to investigate further, but I've run out of ideas here.