Loosing TCP/IPv4 connections with jails+pf on 10.0-RELEASE

Mon Feb 3 09:57:39 UTC 2014

Hello!

We have one server with multiple jails, each jail runs a service (mail,
web, etc.). sysutils/ezjail is used to setup and start the jails.

Beside the public IP address, IPv4 and IPv6 aliases are added to the
main NIC (em0); one per jail. The server has a second NIC (em1) which is
unused.

As we only have one public IPv4 address, pf is used to
    o  redirect connections to jails
    o  NAT connections from jails

With 8.3-RELEASE on another server, this setup was working without
problem. Now that we switched to a new server and 10.0-RELEASE (we
skipped 9.x), we see that TCP connections to jails over IPv4 are having
troubles:

    o  After around 10 days of uptime, connections from an IRC client
       on the host (not a jail) connected to an IRC server on a jail
       are getting dropped during the night (maybe because of no
       activity on the IRC channel). It seems that packets from the
       host (or a remote computer) to the jail are fine. However,
       packets from the jail never reach the peer. This was tested with
       nc(1) on both sides, so the uptime of the IRC client or server
       isn't related.

    o  As the time passes, connections are dropped faster and faster:
       even during the day, when there's activity on the IRC channel.

    o  At some point, connections only live for a few seconds and this
       affects short-lived connections to the SMTP/IMAP and web jails.

A reboot solves the problem, until it comes back a week or more later.
Troubles start to appear again since this week-end.

IPv6 connections are NOT affected: they work perfectly.

This is stock FreeBSD 10.0-RELEASE amd64 with GENERIC kernel.

You'll find attached the output of ifconfig(8), our pf rules and one
jail configuration in ezjail (other jails have a similar setup). Note
that the pf rules we used on FreeBSD 8.3 are commented out at the end of
pf.conf; we simplified them by using ports lists.

Do you see something wrong with this setup?

PS: I'm not subscribed to the list, please CC me.

-- 
Jean-Sébastien Pédron
-------------- next part --------------
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:25:90:21:04:2c
        inet $PUBLIC_IP netmask 0xffffff00 broadcast $PUBLIC_BROADCAST 
        inet6 fe80::225:90ff:fe21:42c%em0 prefixlen 64 scopeid 0x1 
        inet6 $PUBLIC_IPV6::1 prefixlen 56 
        inet 10.0.0.1 netmask 0xffffffff broadcast 10.0.0.1 
        inet 10.0.0.3 netmask 0xffffffff broadcast 10.0.0.3 
        inet6 $PUBLIC_IPV6::1:3 prefixlen 64 
        inet 10.0.0.4 netmask 0xffffffff broadcast 10.0.0.4 
        inet6 $PUBLIC_IPV6::1:4 prefixlen 64 
        inet 10.0.0.2 netmask 0xffffffff broadcast 10.0.0.2 
        inet6 $PUBLIC_IPV6::1:2 prefixlen 64 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:25:90:21:04:2d
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
        inet 127.0.0.1 netmask 0xff000000 
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
-------------- next part --------------
# Interface declarations
ext_if="em0"
int_if="lo0"
all_if="{em0, lo0}"

# Internal network subnet
jail_net="10.0.0.0/24"

# Name and IP of our webserver
MYSQL="10.0.0.1"
HTTPD="10.0.0.2"
VEXIM="10.0.0.3"
IRCD="10.0.0.4"

PUBLIC_IP="..."

#scrub in all

nat pass on $ext_if inet from $jail_net to any -> $PUBLIC_IP

rdr pass on $all_if inet proto tcp from any to $ext_if port {6667,6668,7000} -> $IRCD
rdr pass on $all_if inet proto tcp from any to $ext_if port {80,443,8140} -> $HTTPD
rdr pass on $all_if inet proto tcp from any to $ext_if port {25,143,465,993,995} -> $VEXIM
rdr pass on $int_if inet proto tcp from any to $int_if port 25 -> $VEXIM

### OLD RULES (FreeBSD 8.3) ###

#rdr on $all_if inet proto tcp from any to $ext_if port 80 -> $HTTPD port 80
#rdr on $all_if inet proto tcp from any to $ext_if port 443 -> $HTTPD port 443
#rdr on $all_if inet proto tcp from any to $ext_if port 8140 -> $HTTPD port 8140
#rdr on $all_if inet proto tcp from any to $ext_if port 995 -> $VEXIM port 995
#rdr on $all_if inet proto tcp from any to $ext_if port 993 -> $VEXIM port 993
#rdr on $all_if inet proto tcp from any to $ext_if port 143 -> $VEXIM port 143
#rdr on $all_if inet proto tcp from any to $ext_if port 25 -> $VEXIM port 25
#rdr on $all_if inet proto tcp from any to $ext_if port 465 -> $VEXIM port 465
#rdr on $all_if inet proto tcp from any to $int_if port 25 -> $VEXIM port 25
#rdr on $all_if inet proto tcp from any to $ext_if port 7000 -> $IRCD port 7000
#rdr on $all_if inet proto tcp from any to $ext_if port 6667 -> $IRCD port 6667
#rdr on $all_if inet proto tcp from any to $ext_if port 6668 -> $IRCD port 6668

#nat on $ext_if inet from $MYSQL to any -> $PUBLIC_IP
#nat on $ext_if inet from $HTTPD to any -> $PUBLIC_IP
#nat on $ext_if inet from $VEXIM to any -> $PUBLIC_IP
#nat on $ext_if inet from $IRCD to any -> $PUBLIC_IP
-------------- next part --------------
export jail_ircd_hostname="ircd"
export jail_ircd_ip="em0|10.0.0.4,em0|$PUBLIC_IPV6::1:4"
...
export jail_ircd_parameters="allow.raw_sockets=1"
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 261 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20140203/f31922ae/attachment.sig>