strange ARP problem

[ray at redshift.com]

At 08:59 PM 3/17/2006 -0800, Glenn Dawson wrote:
| At 08:34 PM 3/17/2006, ray at redshift.com wrote:
| >I'm having a strange issue here and thought maybe someone on this list might
| >have some ideas.  I have tried to figure it out for a couple of days, but no
| >luck yet.  The problem seems to be around reporting of arp information.
| >
| >Here is my basic config.  I have my workstation (a windows XP box) with 2 IP's
| >on a private network segment (both with /24 subnet masks)
| >
| >192.168.10.250
| >192.168.20.250
| >
| >the 10.250 and 20.250 are connected out to a small switch. Also connected to
| >that small switch is a mail server as shown below.
| >
| >[ workstation  ]                         [ mail server  ]
| >[192.168.10.250]-------[ small  ]--------[ 192.168.10.15]
| >[192.168.20.250]-------[ switch ]--------[ 192.168.20.15]
| >                            |
| >                            |
| >                   [router 192.168.10.1]
| >                            |
| >                        public IP
| >
| >10.15 handles SMTP to the public, 20.15 is for admin and POP to/from the
| >workstation on 20.250
| >
| >Okay, so the problem is that when I fire up the Workstation (it's running
| >Windows XP), the arp data for 192.168.20.15 comes back with the incorrect Mac
| >address.  It ends up with the Mac address for 10.15, instead of 20.15 - which
| >keeps the machines from talking correctly.  If you delete the ARP table and
| >re-arp, then it's perfectly fine from then on.  Totally odd.
| >
| >Then the other night I noticed the following errors (see below) from the mail
| >server.  It seems to be related, but I can't pin point the source or 
| >what might
| >cause something like this.
| >
| >Does anyone have any ideas what could be causing this?
| >
| > > arp: 192.168.10.1 is on fxp0 but got reply from 00:30:48:52:08:03 on bge0
| > > arp: 192.168.20.250 is on bge0 but got reply from 00:e0:81:32:e0:a0 on fxp0
| > > arp: 00:30:48:51:ce:f0 is using my IP address 192.168.20.15!
| > > arp: 00:30:48:51:ce:f0 is using my IP address 192.168.20.15!
| > > arp: 192.168.10.1 is on fxp0 but got reply from 00:30:48:52:08:03 on bge0
| > > arp: 00:30:48:51:ce:f0 is using my IP address 192.168.20.15!
| > > arp: 192.168.10.15 is on lo0 but got reply from 00:30:48:51:ce:f0 on bge0
| > > arp: 192.168.10.1 is on fxp0 but got reply from 00:30:48:52:08:03 on bge0
| > > arp: 192.168.10.15 is on lo0 but got reply from 00:30:48:51:ce:f0 on bge0
| > > arp: 192.168.20.250 is on bge0 but got reply from 00:e0:81:32:e0:a0 on fxp0
| > > arp: 192.168.10.15 is on lo0 but got reply from 00:30:48:51:ce:f0 on bge0
| > > arp: 192.168.10.1 is on fxp0 but got reply from 00:30:48:52:08:03 on bge0
| >
| >here is the ifconfig from the mail server:
| >
| >[ray at mail ray]$ ifconfig
| >fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
| >         inet 192.168.10.15 netmask 0xffffff00 broadcast 192.168.10.255
| >         ether 00:30:48:51:ce:f0
| >         media: Ethernet autoselect (100baseTX <full-duplex>)
| >         status: active
| >bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
| >         options=1b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING>
| >         inet 192.168.20.15 netmask 0xffffff00 broadcast 192.168.20.255
| >         ether 00:30:48:51:ce:f1
| >         media: Ethernet autoselect (100baseTX <full-duplex>)
| >         status: active
| >lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
| >         inet 127.0.0.1 netmask 0xff000000
| >
| >If anyone has any idea, please let me know.  Thanks!
| 
| This is exactly why it's ill-advised to have two network interfaces 
| on different networks connected to the same physical network.
| 
| If you actually need two different networks (although from your 
| description I don't see a reason why you would) then use a single 
| physical interface and assign it an IP from each network.  Or, get a 
| switch that has VLAN capabilities and keep the two networks separated.
| 
| -Glenn

Someone mentioned (off-list) that it was probably a broadcast issue.  That makes
sense.  If the ARP is doing a broadcast and if that is occurring over layer 2,
would the packets arrive at both 10.x and at 20.x and anything attached
physically to the ports of the switch?

If that's the case, then wouldn't the OS just ignore broadcast requests which
didn't original from it's local network segment (as defined by the subnet?)  Or
does something occurring on layer 2 bypass all the IP layer and work directly
with what is physically attached to the device involved?

Ray