netgraph arp issues vs linux veth

[David Yeske]

I made another attempt with netgraph and I think I'm almost there, but I'm
still having some issues.  I found a linux solution called veth
http://www.geocities.com/nestorjpg/veth/ which might do the job, but I would
prefer to use netgraph if possible.  Here is some more detailed config
information.

I ran this on the spoof machine

# ngctl mkpeer . eiface hook ether
# ifconfig ngeth0 link 00:bd:03:12:12:12
# ifconfig ngeth0 192.168.10.3 netmask 255.255.255.0

# ngctl mkpeer ngeth0: bridge lower link0
# ngctl name ngeth0:lower broken
# ngctl connect fxp0: broken: lower link1
# ngctl connect fxp0: broken: upper link2
# ngctl connect ngeth0: broken: upper link3
# ngctl msg ngeth0: setpromisc 1
# ngctl msg ngeth0: setautosrc 0
# ngctl msg fxp0: setpromisc 1
# ngctl msg fxp0: setautosrc 0

# ngctl show broken:
  Name: broken          Type: bridge          ID: 00000046   Num hooks: 4
  Local hook      Peer name       Peer type    Peer ID         Peer hook
  ----------      ---------       ---------    -------         ---------
  link3           ngeth0          ether        00000005        upper
  link2           fxp0            ether        00000004        upper
  link1           fxp0            ether        00000004        lower
  link0           ngeth0          ether        00000005        lower

on the remote machine an arp -a lists this
? (192.168.10.3) at 00:bd:03:12:12:12 on rl0 [ethernet] 
? (192.168.10.1) at 00:00:e8:5b:13:44 on rl0 permanent [ethernet]

on the spoof machine an arp -a lists this
? (192.168.10.1) at (incomplete) on ngeth0 [ethernet]
? (192.168.10.3) at 00:bd:03:12:12:12 on ngeth0 permanent [ethernet]

a sniff on the spoof machine listed this while pinging the remote machine

# tcpdump -i ngeth0 'ether host 00:00:e8:5b:13:44'
tcpdump: listening on ngeth0
14:03:30.519263 arp reply 192.168.10.1 is-at 0:0:e8:5b:13:44
14:03:33.416568 192.168.10.1 > 192.168.10.3: icmp: echo request
14:03:40.530562 arp reply 192.168.10.1 is-at 0:0:e8:5b:13:44
14:03:43.427175 192.168.10.1 > 192.168.10.3: icmp: echo request
14:03:50.540805 arp reply 192.168.10.1 is-at 0:0:e8:5b:13:44
14:03:53.437845 192.168.10.1 > 192.168.10.3: icmp: echo request
14:04:00.550960 arp reply 192.168.10.1 is-at 0:0:e8:5b:13:44
14:04:03.448383 192.168.10.1 > 192.168.10.3: icmp: echo request

a sniff on the remote machine listed this while pinging the spoof machine

# tcpdump -i rl0 'ether host 00:bd:03:12:12:12'
tcpdump: listening on rl0
14:02:24.918804 192.168.10.1 > 192.168.10.3: icmp: echo request
14:02:29.179263 arp reply 192.168.10.1 is-at 0:0:e8:5b:13:44
14:02:34.929051 192.168.10.1 > 192.168.10.3: icmp: echo request
14:02:44.939136 192.168.10.1 > 192.168.10.3: icmp: echo request
14:02:52.052260 arp reply 192.168.10.1 is-at 0:0:e8:5b:13:44
14:02:54.949402 192.168.10.1 > 192.168.10.3: icmp: echo request
14:03:02.063079 arp reply 192.168.10.1 is-at 0:0:e8:5b:13:44
14:03:04.959534 192.168.10.1 > 192.168.10.3: icmp: echo request
14:03:12.072830 arp reply 192.168.10.1 is-at 0:0:e8:5b:13:44

Any clues or pointers are greatly appreciated and will mean I get to deploy
FreeBSD with netgraph rather than linux with veth.

Regards,
David Yeske