Jails and IP Aliasing

[David Allen]

Unless I'm losing my mind, I'm encountering what seems to yet another
gotcha with jails.  The following has been dumbed down for clarity and
brevity.

---------------------------------------------------------------------
# hostname
jailhost.example.org

# host jailhost
jailhost.example.org has address 10.0.1.2

# ifconfig fxp0
fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=b<RXCSUM,TXCSUM,VLAN_MTU>
        ether 00:07:e9:c8:2e:32
        inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255
        inet 10.0.1.3 netmask 0xffffffff broadcast 10.0.1.3
        inet 10.0.1.4 netmask 0xffffffff broadcast 10.0.1.4
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active

# grep jail /etc/rc.conf
...
jail_ns_hostname="ns.example.org"
jail_ns_ip="10.0.1.3"
...
jail_mail_hostname="mail.example.org"
jail_mail_ip="10.0.1.4"

# sockstat -4l
USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS
root     sendmail   11556 4  tcp4   10.0.1.4:25           *:*
root     syslogd    10591 6  udp4   10.0.1.4:514          *:*
root     sendmail   10438 4  tcp4   10.0.1.3:25           *:*
bind     named      4011  20 udp4   10.0.1.3:53           *:*
bind     named      4011  21 tcp4   10.0.1.3:53           *:*
bind     named      4011  22 tcp4   10.0.1.3:953          *:*
root     syslogd    897   6  udp4   10.0.1.3:514          *:*
root     sshd       715   3  tcp4   10.0.1.2:22           *:*
root     syslogd    563   6  udp4   127.0.0.1:514         *:*
root     sendmail   489   4  tcp4   127.0.0.1:25          *:*

---------------------------------------------------------------------

If I telnet from the jailhost to mail.example.org 25, for example, both
outgoing and incoming connections appear to sockstat, tcpdump, etc. on the
jailhost as using the jail's IP address!  Similarly, if I perform a DNS
lookup on the jailhost (using the ns.example.org jail for resolution),
both incoming and outgoing connections occur on the jail's IP address.

Granted, everything is really happening over the loopback address, but a
connection originating from the jailhost to a jail should appear to be
using the jailhost's IP address, or so I'd like to think.  If it doesn't,
then the scenario is awkward at best when trying to understand or debug
issues.

The thought occurred to me, however, that I could add a new network card
and reserve that for the IP aliases needed by the jails.  But I'm not sure
whether that will work in telling me who's who, or whether I'll discover
another gotcha.  ;-)

Comments, questions and complaints all welcomed.