Jails, pf and larger packets

[Nolan J. Darilek]

I'm having a bit of a problem with jails on my FreeBSD 5.2.1
system. Specifically, since I switched from my ipfirewall/natd-based
firewall to pf, jails can't seen to send packets of greater than 46
bytes of data to other systems.

My network is set up as follows. The firewall/host box, running 5.2.1,
is on 192.168.0.1. On 192.168.0.2 I have a jail running numerous
services, and on 192.168.0.3 I have a shell account jail for screen
sessions and such. The internal LAN is also served by DHCP, and my
desktop Debian box is at 0.100.

The problem first surfaced when I was using esd on the shell server to
stream and play audio on my desktop for IM events. I ran pftcpdump on
pflog0 but saw no messages about blocked packets. Though I can't
traceroute out of the jail to see where my packets are going,
traceroutes from my desktop seem to indicate that jails appear to
bridge, so it seems odd that anything on the firewall could have any
effect on traffic within the LAN to the jail. I then ran strace on the
esdplay process, and received the following edited output:
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 4
fcntl(4, F_SETFD, FD_CLOEXEC) = 0
setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [0], 4) = 0
connect(4, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, 16)
= 0
. . .
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
write(4, "\0\0\0\0", 4) = 4
read(4, "\0\0\0\0", 4) = 4
. . .
write(4, "\0\0\0\0", 4) = 4
write(4, "\0\0\0\0", 4) = 4
write(4, "\0\0\0\0", 4) = 4
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128) =
128
setsockopt(4, SOL_SOCKET, SO_SNDBUF, [0], 4) = 0
setsockopt(4, SOL_SOCKET, SO_RCVBUF, [0], 4) = 0
. . .
write(4, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) =
-1 EPERM (Operation not permitted)
close(4) = 0
. . .

Could a firewall cause a permission denied error like this? I'd think
not, but I don't know much about networking, and the only major change
I recall making was switching from ipfirewall/natd to pf for
everything (which, incidentally, works perfectly except for this
issue.)

I then tried to create a simple test case, trying different
permutations of that. I wrote a simple client and server. The server
binds to the IP address and simply listens for a single string, and
the client sends a string of the specified length. I can send data TO
the jails from my desktop just fine, but I can reproduce the EPERM
when sending data from jails to the desktop. Interestingly enough, the
error begins when I send packets of 47 bytes or greater. It's been
awhile since I've done TCP/IP; does the header happen to be 18 bytes?
If so, is there something preventing packets of greater than 64 bytes
from leaving jails? I see nothing in logs and have consulted the
manpages, though it's certainly possible that I've missed something.

And, again, this worked fine before I switched to pf. I'd rather not
switch back to ipfirewall if I can avoid it, as I hope to use queueing
soon and, in general, like the look of pf rules a bit more. Is there
anything else that might be responsible for this failure, and what
else can I offer to help you help me?

Thanks.