udp - weird behavior of reply-to

Tue Jan 10 08:29:02 UTC 2017

On Mon, Jan 09, 2017 at 10:01:21PM -0500, Ian FREISLICH wrote:
> On 01/09/17 17:17, Marek Zarychta wrote:
> > On Mon, Jan 09, 2017 at 09:58:38PM +0100, Kristof Provost wrote:
> >> On 9 Jan 2017, at 18:25, Marek Zarychta wrote:
> >>> On Sun, Jan 08, 2017 at 07:08:10PM +0100, Kristof Provost wrote:
> >>>> On 8 Jan 2017, at 15:55, Marek Zarychta wrote:
> >>>> The problem description doesn’t ring any bells with me, but I’m
> >>>> also
> >>>> not sure > >> I’ve fully understood it.  Can you document a minimal reproduction
> >>>> scenario,
> >>>> with a pf.conf and perhaps network captures documenting the problem?
> >>>>
> >>>> There’s certainly not been a conscious decision to break UDP
> >>>> reply-to.
> >>>>
> >>> Let me apologize, the problem wasn't previously properly identified.
> >>> It
> >>> seems to be more problem of UDP protocol implementation than PF issue.
> >>> UDP sockets are opened and bound to address of the outgoing interface
> >>> (interface which has a route to the client). Because the socket is not
> >>> bound to the incoming interface, the PF reply-to rules couldn't be
> >>> evaluated.  By the way, TCP sockets are bound to the interface where
> >>> the
> >>> traffic arrives and everything works fine.
> >>> This machine is i386 running 11.0-STABLE r311772
> >>>
> >>> The problem remains unresolved. Are there any corresponding sysctls
> >>> correcting this behavior and enabling the opportunity to use PF
> >>> assisted
> >>> symmetric routing scenario again?
> >>>
> >> How are your UDP listen sockets set up?
> >> Are they bound to a specific interface, or are they listening on
> >> 0.0.0.0?
> > Yes, socket is listening on 0.0.0.0, the client from outside network  is
> > initiating  connection and initiating packet arrives on interface B,
> > which is supposed only to communicate with devices on its own network
> > (no additional routes go via this interface), so normally the reply
> > would be passed via interface A having default gateway in scope and
> > communication would fail.
> > With the assistance of PF reply-to rule, TCP services are able to pass
> > reply from interface B via other, second gateway:  reply-to (B GW2).
> 
> Are you saying that your network looks approximately like this and there 
> is no route to the client network where X resides on your server:
> 
> iface-A--<default route>--GW1
> iface-B--_local network_--GW2--_client X_
> 
> Client X originates a UDP "connection" to B and that return traffic to X 
> leaves interface A despite your reply-to rule.
> 
> I would be very interested to know:
> 
> 1. whether the reply-to rule actually matches on the inbound traffic.  
> You can make the rule log and tcpdump on the pflog0 interface.  I 
> believe the -e option to tcpdump will show the rule that matched.
> 
> 2. the output of "pfctl -s sta |grep IP_of_X"
> 
> 3. what software you're using for your UDP server.  I can try to 
> reproduce your issue.
> 
> Ian
Both interfaces have public IP addresses,  server has a route to client
via GW1, client originates connection to B via GW2. TCP replies
originate from socket associated with (bound to) interface B so reply-to
works as expected and client is able to connect to both interfaces
simultaneously. Corresponding PF states are shown below
all tcp ADDR_A:22 <- CLIENT_ADDR:57598       ESTABLISHED:ESTABLISHED
all tcp ADDR_B:22 <- CLIENT_ADDRR:58856       ESTABLISHED:ESTABLISHED

UDP datagrams never originate from socket B ignoring client effort to
use socket associated with interface B, what was investigated using
sockstat. Server always tries to respond via interface A and simply
create new PF state as shown below. 
all udp ADDR_B:1194 <- CLIENT_ADDRR:35607       NO_TRAFFIC:SINGLE
all udp ADDR_A:1194 -> CLIENT_ADDRR:35607       SINGLE:NO_TRAFFIC
This is also seen by pflog:
06:38:52.195266 rule 487/0(match): pass in on B: CLIENT_ADDRR.35607
> ADDR_B.1194: UDP, length 42
06:38:52.196970 rule 488/0(match): pass out on A: ADDR_A.1194 >
CLIENT_ADDRR.35607: UDP, length 54

Rules 487 and 488 are shown below:

 pass in log quick on $B \
       reply-to ( $B $GW2 ) \
       inet proto udp \
       from any \
       to ($ext_if_2:0) port $udp_services \
       keep state

 pass out log quick on $A proto
       udp \
       from any to any \
       keep state

Successful setup has been previously used to allow client X use address
of interface B as OpenVPN remote address. The scenario could be
reproduced simply using netcat as a server and client: nc -u -l 1194,
nc -u ADDR_B 1194.   
> 
> > This functionality is currently broken for any UDP service, because UDP
> > sockets are always opened on supposed_to_be_outgoing interface A and
> > bound to the address of this interface, which is considered good from
> > routing table perspective, but silently breaks PF reply-to for UDP.
> >
> > When the machine was running 9-STABLE reply-to had successfully been
> > used to assist both TCP and UDP driven services.
> >
> > Is anyone reading this list still using reply-to rule for routing UDP
> > traffic back via incoming interface?
> >
> > Maybe currently, the better place to discuss this questions would be
> > freebsd-net, but the thread was initiated here.
> >
> >> I’m afraid I’m still struggling to understand what your setup is,
> >> what you’re
> >> expecting to see and what you’re seeing instead.
> >>
> >> Regards,
> >> Kristof
> >
> >
> 
> 
> -- 
>  
> 
> Cape Augusta Digital Properties, LLC a Cape Augusta Company
> 
> *Breach of confidentiality & accidental breach of confidentiality *
> 
> This email and any files transmitted with it are confidential and intended 
> solely for the use of the individual or entity to whom they are addressed. 
> If you have received this email in error please notify the system manager. 
> This message contains confidential information and is intended only for the 
> individual named. If you are not the named addressee you should not 
> disseminate, distribute or copy this e-mail. Please notify the sender 
> immediately by e-mail if you have received this e-mail by mistake and 
> delete this e-mail from your system. If you are not the intended recipient 
> you are notified that disclosing, copying, distributing or taking any 
> action in reliance on the contents of this information is strictly 
> prohibited.
> _______________________________________________
> freebsd-pf at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-pf
> To unsubscribe, send any mail to "freebsd-pf-unsubscribe at freebsd.org"
> 

-- 
Marek Zarychta
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-pf/attachments/20170110/55129816/attachment.sig>