Issue with routing table entries, jails and pf filtering on loopback interfaces
Thomas Steen Rasmussen
thomas at gibfest.dk
Fri Jun 19 07:11:19 UTC 2015
-----BEGIN PGP SIGNED MESSAGE-----
This will be a long post, apologies, but it is a complex issue. First I
will explain how the server is configured, then I will explain the
problem and the workaround I found.
When I add one or more IP aliases to a non-loopback interface, lagg0 in
this example, FreeBSD adds two routing table entries per IP:
22.214.171.124 link#6 UHS lo0
126.96.36.199/32 link#6 U lagg0
The first entry, the one with "lo0" as it's interface is the problem.
The server also has an extra loopback interface with rfc1918 addresses
for jails that do not need a real, public IP. A pretty common setup.
I also run pf on the server, first filtering rule is "block log all" so
I explicitly need to permit traffic on every interface (I do
"set skip on lo0" though). I permit all outgoing traffic both on
lo1 and on lagg0, so only incoming traffic needs to be permitted.
The problem is that with this setup an rfc1918 jail _cannot_ establish a
TCP connection to a service listening on an IP on the lagg0 interface.
I think this is because the "lo0" routing table entry forces the traffic
through an interface where it logically should not appear.
When I try to ssh from a jail with an rfc1918 IP (say 10.0.0.1) on lo1
to a jail with IP 188.8.131.52 on lagg0, then pf blocks the traffic,
but not as I'd expect. I'd expect it to block an incoming packet on
lagg0 from the rfc1918 ip to the jailhosts IP. Instead it blocks the
syn/ack packet, but in the wrong direction, check this tcpdump from pflog0:
06:51:38.188170 rule 0..16777216/0(match): block out on lo1:
184.108.40.206.22 > 10.0.0.1.39228: Flags [S.], seq 573771477, ack
2565048197, win 65535, options [mss 16344,nop,wscale 6,sackOK,TS val
3503997415 ecr 53572243], length 0
This traffic should never appear in this direction on this interface. No
wonder it gets blocked.
I've been able to workaround this issue by deleting the "lo0" routes for
the IPs on lagg0. If I try SSH again everything looks correct again:
06:57:44.590356 rule 0..16777216/0(match): block in on lagg0:
10.0.0.1.48211 > 220.127.116.11.22: Flags [S], seq 3762725019, win 65535,
options [mss 1460,nop,wscale 6,sackOK,TS val 53938646 ecr 0], length 0
The syn packet is blocked where I'd expect it to get blocked, and if I
permit 10.0.0.1 to SSH to 18.104.22.168 in pf.conf it works as expected.
I've been asking around and it seems like I'm not the only one with this
problem. People just do "set skip on lo1" and never notice it. From a
security perspective though I really don't want traffic between jails
to be unfirewalled.
Can anyone shine a light on this? Thanks a bunch in advance :)
Thomas Steen Rasmussen
ps. For what it's worth OpenBSD adds an lo0 route for IP aliases on
real interfaces too. No idea if the same problem is present though.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (MingW32)
-----END PGP SIGNATURE-----
More information about the freebsd-net