Jails and IP Aliasing

[David Allen]

On Mon, Jul 7, 2008 at 10:54 AM, Jason Morgan
<jwm-freebsd-questions at sentinelchicken.net> wrote:
> On 2008.07.07 09:51:33, David Allen wrote:
>> Unless I'm losing my mind, I'm encountering what seems to yet another
>> gotcha with jails.  The following has been dumbed down for clarity and
>> brevity.
>>
>> ---------------------------------------------------------------------
>> # hostname
>> jailhost.example.org
>>
>> # host jailhost
>> jailhost.example.org has address 10.0.1.2
>>
>> # ifconfig fxp0
>> fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         options=b<RXCSUM,TXCSUM,VLAN_MTU>
>>         ether 00:07:e9:c8:2e:32
>>         inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255
>>         inet 10.0.1.3 netmask 0xffffffff broadcast 10.0.1.3
>>         inet 10.0.1.4 netmask 0xffffffff broadcast 10.0.1.4
>>         media: Ethernet autoselect (100baseTX <full-duplex>)
>>         status: active
>
> This is the output for my jail interface. Notice that your jail
> aliases are broadcasting on the jail's IP. I don't know if this is an
> issue or not (my jails run on i386 FBSD 6.3), but it's something to
> look at. How are you setting the aliases?
>
> sk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>     options=b<RXCSUM,TXCSUM,VLAN_MTU>
>     inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
>     inet 10.0.0.101 netmask 0xffffff00 broadcast 10.0.0.255
>     inet 10.0.0.201 netmask 0xffffff00 broadcast 10.0.0.255
>     ether xx:xx:xx:xx:xx:xx
>     media: Ethernet autoselect (1000baseTX <full-duplex,flag0,flag1>)
>     status: active

My own aliases:

# grep fxp0 /etc/rc.conf
ifconfig_fxp0="inet 10.0.1.2 netmask 0xffffff00"
ifconfig_fxp0_alias0="10.0.1.3 netmask 0xffffffff"
ifconfig_fxp0_alias1="10.0.1.4 netmask 0xffffffff"
ifconfig_fxp0_alias2="10.0.1.5 netmask 0xffffffff"

My understanding from the handbook is that the mask should be set to all
ones if the alias is for an address that's part of the same network.  For
a different segment, it's the first alias that should be set to the real
netmask, with any additional aliases using a netmask of all ones.

Granted, the broadcast addresses looks odd.  If I my programming skills
were better, I'd just read through the code and understand what's really
happening, but for now, I'm just taking the FreeBSD folks at their word at
following instructions.  That's a roundabout way of saying I think your
aliases are set up incorrectly.  ;-)

If you're not seeing the behaviour I'm seeing, do let me know.  But to
clarify with a concrete example, the following is what I see on the
jailhost (10.0.1.2) when it connects to port 25 on one of the
jails (10.0.1.5).

# tcpdump -nqti lo0 port 25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo0, link-type NULL (BSD loopback), capture size 96 bytes
IP 10.0.1.5.62110 > 10.0.1.5.25: tcp 0
IP 10.0.1.5.25 > 10.0.1.5.62110: tcp 0
IP 10.0.1.5.62110 > 10.0.1.5.25: tcp 0
IP 10.0.1.5.25 > 10.0.1.5.62110: tcp 89
IP 10.0.1.5.62110 > 10.0.1.5.25: tcp 0

# netstat -nf inet
Active Internet connections
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp4       0      0  10.0.1.5.25            10.0.1.5.62110         ESTABLISHED
tcp4       0      0  10.0.1.5.62110         10.0.1.5.25            ESTABLISHED

# sockstat -4 -p 25
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sendmail   16594 1  tcp4   10.0.1.5:25           10.0.1.5:62110
root     sendmail   16594 4  tcp4   10.0.1.5:25           10.0.1.5:62110
root     sendmail   16594 7  tcp4   10.0.1.5:25           10.0.1.5:62110
root     telnet     16593 3  tcp4   10.0.1.5:62110        10.0.1.5:25

Why the jailhost is suddenly using the jail's IP address is beyond me.