kludgily solved: bridge with access on both interfaces

[Ian Smith]

On Thu, 25 Dec 2003, Bruce A. Mah wrote:

 > If memory serves me right, Ian Smith wrote:
 > 
 > > In short, ifconfig appears unwilling to have two NICs covering the same
 > > /24.  Can this be set up?  I'm also at a bit of a loss with the routing,
 > > so inside packets to the bridge box (ie unbridged packets) are responded
 > > to on the same interface, and outside unbridged packets go only to/from
 > > the gw.  Some tcpdumps on both in and outside interfaces suggest an ARP
 > > response problem also, perhaps; no responses on the inside iface at all.
 > 
 > Hi Ian--

Hi Bruce, and thanks also to Michael and Robert for contributions; the
Soekris-symptoms thread was of interest too - I want one someday ..

 > This may or may not be the source of your problem, but I've been
 > procrastinating on writing this up for a couple months and this was
 > the impetus that pushed me over the edge.

Always glad in helping to push someone over the edge!

Seems likely the problem, but I guess working around it we've come up
with what seems a bizarre solution with routes, that does work to put an
address on each interface that is only accessible (via ARP, I assume) 
from its own interface side of the bridge - which is actually what we
wanted, though I still don't really understand why it works!

 > In 4-STABLE, there's a bug that prevents ARP from working correctly on
 > unnumbered bridge interfaces when bridging is enabled using the
 > bridge.ko module.  Basically, there are some checks in the ARP code
 > that decide when to accept inbound ARP packets.  These checks are a
 > little different when the inbound interface is part of a bridge group.

Ah, so.  I'd spent some time playing with 'pub' and 'pub only' entries
trying to get around this behaviour before reading your message, but
wound up having better luck with route, though it was still a struggle
despite the copious examples of syntax usage in man 8 route :-)

 > Some of these tests are conditional on the BRIDGE preprocessor symbol;
 > this symbol gets defined if "options BRIDGE" is compiled into the
 > kernel but not if you use the bridge.ko module.  As a result, ARP
 > packets on unnumbered interfaces get thrown away.

Or ones on numbered interfaces, but on the wrong/other side, it seems.

 > The workaround for this problem is just to compile BRIDGE into the
 > kernel.  Manuel Kasper and I spent a few cycles working on this trying
 > to make a m0n0wall box into a filtering bridge.

Always happy to follow already beaten paths!

 > For more specifics, see src/sys/netinet/if_ether.c (grep for BRIDGE in
 > this file).

I guess I wondered why it wouldn't check the sysctl to see if bridging's
enabled, but I gather light weight's needed here, and runtime's unknown.

Anyway, maybe the following will break with BRIDGE in the kernel (next
job after our current heatwave abates!) and maybe the single IP for the
bridge box that everyone has advised is best will then be accessible on
either interface - which would be fine too, I'm happy to differentiate
outside and inside access to services by IPFW - but here's what works,
apparently reliably so far, while still using 4.8-RELEASE GENERIC:

/etc/rc.conf:

  ifconfig_ed0="inet aaa.bbb.c.174  netmask 255.255.255.248"
  ifconfig_ed1="inet aaa.bbb.c.173  netmask 255.255.255.255"
  defaultrouter="aaa.bbb.c.171"

/etc/rc.local:

  kldload bridge
  sysctl net.link.ether.bridge_cfg=ed0,ed1
  sysctl net.link.ether.bridge=1

  # yes I know the syntax is weird, and the last is a netmask
  route add aaa.bbb.c.169 aaa.bbb.c.173 -interface aaa.bbb.c.173
  # and nothing worked (on reboot) even with that, until this!
  route delete -net aaa.bbb.c.169/32

smithi on tubi% arp -an
? (aaa.bbb.c.169) at 00:aa:00:b7:6c:1d on ed1 [ethernet]
? (aaa.bbb.c.171) at 00:80:48:9e:0b:db on ed0 [ethernet]
? (aaa.bbb.c.173) at 52:54:05:e3:d9:a5 on ed1 permanent [ethernet]
? (aaa.bbb.c.174) at 52:54:05:e4:28:3d on ed0 permanent [ethernet]
? (aaa.bbb.c.175) at ff:ff:ff:ff:ff:ff on ed0 permanent [ethernet]

Now .174 is only visible from ed0, and .173 from ed1 (as desired/ok),
and .169, our mock 'outside' gateway in this setup, is accessible (as
well as being properly bridged to all the 'inside' hosts on ed0).

smithi on tubi% netstat -finet -ran
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            aaa.bbb.c.171      UGSc        1        0    ed0
127.0.0.1          127.0.0.1          UH          1     1206    lo0
aaa.bbb.c.131      aaa.bbb.c.171      UGHW3       0       47    ed0 417
aaa.bbb.c.168/29   link#1             UC          3        0    ed0
aaa.bbb.c.169      00:aa:00:b7:6c:1d  UHLW        1      654    ed1 864 =>
aaa.bbb.c.169&0xcb2934ad link#2       UCSc        1        0    ed1
aaa.bbb.c.171      link#1             UHLW        2       61    ed0
aaa.bbb.c.173      52:54:05:e3:d9:a5  UHLW        0      951    lo0 =>
aaa.bbb.c.173/32   link#2             UC          1        0    ed1
aaa.bbb.c.174      52:54:05:e4:28:3d  UHLW        0        0    lo0
aaa.bbb.c.175      ff:ff:ff:ff:ff:ff  UHLWb       3     1482    ed0

Seems from my reading to date that a correct single route command using
'-interface ed1' correctly, possibly with the -ifp (?) modifier in the
right place (?), may give the same result (ie .173 <==> .169 routed via
ed1, with .174 being its address everywhere on the 'inside' ed0), which
is correct for this test rig - but I can't figure out the syntax; any
pointers to some beyond-the-basics route(8) usage, anyone?

ie, the below is working as the right result, but what's a better way to
get to it than those two route statements above, with the weird netmask?

smithi on tubi% route -vn get aaa.bbb.c.169
u: inet aaa.bbb.c.169; u: link ; RTM_GET: Report Metrics: len 164,
  pid: 0, seq 1, errno 0, flags:<UP,GATEWAY,HOST,STATIC>
locks:  inits:
sockaddrs: <DST,IFP>
 aaa.bbb.c.169
   route to: aaa.bbb.c.169
destination: aaa.bbb.c.169
  interface: ed1
      flags: <UP,HOST,DONE,LLINFO,WASCLONED>
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu    expire
       0         0         0         0         0         0      1500    1171
locks:  inits:
sockaddrs: <DST,GATEWAY,IFP,IFA>
 aaa.bbb.c.169 0.aa.0.b7.6c.1d ed1:52.54.5.e3.d9.a5 aaa.bbb.c.173
  
 > Merry Christmas!

and a Happy and Peaceful New Year!

Ian