TCP conection problems IBM VM -> FreeBSD

[Torbjorn Granlund]

I have the following setup:

IBM VM mainframe <-> [Internet] <->  fbsd 6.2 router   <-> fbsd 4.10 smtp box
    "vm"                             "router.swox.se"       "smtp.swox.se"

When vm attempts to make a TCP connection (e.g., on port 25) to
smtp.swox.se I see the following traffic on the router:

22:46:27.015389 IP vm.se.lsoft.com.47218 > smtp.swox.se.smtp: S 27523124:27523124(0) win 8192 <mss 1420,wscale 0,nop,nop,nop,timestamp 1888741492 0>
22:46:27.015523 IP smtp.swox.se.smtp > vm.se.lsoft.com.47218: S 1745147473:1745147473(0) ack 3530628660 win 57344 <mss 1460>
22:46:27.056277 IP vm.se.lsoft.com.47218 > smtp.swox.se.smtp: R 3530628660:3530628660(0) win 0

I.e., the vm box appears to dislike the SYNACK from smtp.swox.se, and
sends an RST.  One might ask if it is the fault of vm or of smtp.swox.se.

If I switch on "pf", the kernel packet filter, on router.swox.se, using
pfctl -e, with an empty /etc/pf.conf, the router discards the SYNACK
from smtp.swox.se.  I realize that an empty /etc/pf.conf means that a
set of implicit rules are enabled, including some sanitize rules.

This leads me to believing that smtp.swox.se somehow sends a bad
SYNACK reply to vm.  But smtp.swox.se is perfectly capable of
accepting TCP connections from lots of machines out there, and the
router leavs the SYNACKs alone except when vm is on the receiving end.

I have stared at tcpdumps in order to try to understand what might be
wrong, unsuccessfully.  I have not found out how to make pf on
router.swox.se be so kind as to log the reason for that it throws away
the SYNACK.  And I don't have access to nor knowledge to of the vm
system to get any information on why it dislikes the SYNACK.

Making tcp connections in the other direction (smtp.swox.se -> vm)
works flawlessly.

I have tried enabling and disabling rfc1323 on smtp.swox.se, without
any change wrt this issue.

My questions:

  How can I debug this further?

  Was there a tcp bug in 4.10 that might be causing the observed
  behaviour?  (Is it fixed in 4.11?)

(I might be upgrading smtp.swox.se to a much more recent FreeBSD
version at some point, but I'd rather not hurry up with it since this
machine runs a lot of things.  Understanding this problem is important
in any case.)

-- 
Torbjörn