netgraph snooping failing using tcpdump with ng_tee and ng_eiface

[Jeff Kletsky]

I'm in the process of trying to debug a deeper question with netgraph,
but am puzzled as to why I can't seem to use tcpdump with ng_tee and
ng_eiface. I don't see any packets with tcpdump on either the ng_eiface
connected to ng_tee left2right or to ng_tee right2left when there are
packets flowing through the ng_tee.

TL;DR
I can't see packets using tcpdump on ng_eiface connected to ng_tee

The configuration can be seen in detail with a graphic from ncgtl dot:
<http://wildside.wagsky.com/freebsd/ngctl/ngctl.testjail_tapped.png>

In summary:

re0 (ether) ----------\
      |                |
re0_tee_upper    re0_tee_lower
      |                |
re0_bridge -----------/
      |
ng0_testjail_tee
      |
ng0_testjail (eiface, passed to a vnet-enabled jail)

The jail can clearly communicate through ng0_testjail to the outside
world (physically connected to re0)

(ifconfig and netstat -rn for host and jail at the bottom of this message)

I've added ng_eiface nodes to all the left2right and right2left tees:

+ mkpeer ng0_testjail_tee: eiface left2right ether
+ mkpeer ng0_testjail_tee: eiface right2left ether
+ mkpeer re0_tee_lower: eiface left2right ether
+ mkpeer re0_tee_lower: eiface right2left ether
+ mkpeer re0_tee_upper: eiface left2right ether
+ mkpeer re0_tee_upper: eiface right2left ether

If I run 'tcpdump -i ngeth1' on the host (left2right tap on ng_tee
between the jail's VNET ng_eiface and the ng_bridge), I can see it is
put into promiscuous mode:

ngeth1: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
     options=28<VLAN_MTU,JUMBO_MTU>
     ether 00:00:00:00:00:00
     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
     media: Ethernet autoselect (1000baseT <full-duplex>)
     status: active

If I make a connection to the outside world from inside the jail, I
would expect the packets to flow through
   ng0_testjail (eiface in jail)
   ng0_testjail_tee
   re0_bridge
   re0_tee_lower or re0_tee_upper
   re0
and back again.

Based on this, I would expect there to be packets copied to the taps
of the ng0_testjail_tee and then to the ng_eiface tap attached to the 
ng_tee.

However, I don't see anything with tcpdump on the ng_eiface tap.

What am I missing here in being able to "snoop" the traffic within my
virtual netgraph network?

Are the packets somehow bypassing the virtual network and being routed
directly to re0?



TIA,

Jeff







Host:
-----

re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 
mtu 1500
  options=8209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
     ether d0:50:99:51:38:eb
     inet 192.168.6.13 netmask 0xffffff00 broadcast 192.168.6.255
     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
     media: Ethernet autoselect (1000baseT <full-duplex,master>)
     status: active


Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            192.168.6.1        UGS         re0
127.0.0.1          link#2             UH          lo0
192.168.6.0/24     link#1             U           re0
192.168.6.13       link#1             UHS         lo0


VNET jail:
----------

ng0_testjail: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 
0 mtu 1500
     options=28<VLAN_MTU,JUMBO_MTU>
     ether 02:00:28:51:38:eb
     inet 192.168.6.213 netmask 0xffffff00 broadcast 192.168.6.255
     nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
     media: Ethernet autoselect (1000baseT <full-duplex>)
     status: active


Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            192.168.6.1        UGS    ng0_test
127.0.0.1          link#1             UH          lo0
192.168.6.0/24     link#2             U      ng0_test
192.168.6.213      link#2             UHS         lo0

arp -a:

wildside.pn.wagsky.com (192.168.6.1) at 68:05:ca:34:34:7f on 
ng0_testjail expires in 966 seconds [ethernet]
? (192.168.6.213) at 02:00:28:51:38:eb on ng0_testjail permanent [ethernet]