Wrong outgoing interface with multiple routing tables
Oleg Sharoyko
os at sfedu.ru
Mon Jul 27 14:34:50 UTC 2009
Hello!
I'm having a trouble with multiple routing tables (FreeBSD 7.2 release).
Either I'm missing something in my setup or packets for daemons started
with setfib are being sent out via the wrong interface.
What I'd like to implement:
em0 - internal management network with ip address 10.2.5.2/24 and
default route 10.2.5.1
em1 - public interface to be used in jail with ip address
195.208.245.229/27 and default route 195.208.245.225
Here are my routing tables:
r61net-fbsdhost-1, / # setfib -0 netstat -rn
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 10.2.5.1 UGS 0 350 em0
10.2.5.0/24 link#1 UC 0 0 em0
10.2.5.1 00:1e:4a:b4:ea:c0 UHLW 2 0 em0 1182
127.0.0.1 127.0.0.1 UH 0 30 lo0
r61net-fbsdhost-1, / # setfib -1 netstat -rn
Routing tables
Internet:
Destination Gateway Flags Refs Use Netif Expire
default 195.208.245.225 UGS 0 0 em1
195.208.245.224/27 link#2 UC 0 0 em1
195.208.245.225 link#2 UHLW 2 0 em1
Firewall:
r61net-fbsdhost-1, / # ipfw show
00001 0 0 setfib 1 ip from any to any in recv em1
00010 0 0 count ip from any to any dst-port 2222 fib 0
00011 0 0 count ip from any 2222 to any fib 0
00012 0 0 count ip from any to any dst-port 2222 fib 1
00013 0 0 count ip from any 2222 to any fib 1
00100 0 0 allow ip from any to any via lo0
00200 0 0 deny ip from any to 127.0.0.0/8
00300 0 0 deny ip from 127.0.0.0/8 to any
65000 30 2648 allow ip from any to any
65535 0 0 deny ip from any to any
With this setup almost everything works as I expect. For example ICMP
echo requests and responses are being received and sent via em1. Both
when ping runs on this host as "setfib 1 ping other_host" and when other
host pings ip address of em1. Connection attempts (setfib 1 telnet
other_host) are also being sent out of the right interface. But when it
comes to the daemons I run into troubles.
I use sshd for tests (have also tried other daemons with no luck):
r61net-fbsdhost-1, / # setfib 1 /usr/sbin/sshd -o 'ListenAddress 195.208.245.229:2222' -D
sshd is bound only to ip address of em1:
r61net-fbsdhost-1, / # sockstat | grep 2222
root sshd 839 3 tcp4 195.208.245.229:2222 *:*
While doing telnet 195.208.249.229 2222 from another host I got following packet traces:
r61net-fbsdhost-1, / # tcpdump -i em0 port 2222
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 96 bytes
17:39:34.872475 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: S 2590499299:2590499299(0) ack 3939022576 win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 3248254533 147282318>
17:39:34.902622 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 1:41(40) ack 1 win 8326 <nop,nop,timestamp 3248254533 147282318>
17:39:37.572271 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: P 41:60(19) ack 7 win 8326 <nop,nop,timestamp 3248254593 147282585>
17:39:37.572293 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: F 60:60(0) ack 7 win 8326 <nop,nop,timestamp 3248254593 147282585>
17:39:37.572986 IP stat.r61.net.2222 > brain.cc.rsu.ru.49293: . ack 8 win 8325 <nop,nop,timestamp 3248254593 147282585>
r61net-fbsdhost-1, / # tcpdump -i em1 port 2222
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes
17:39:34.872370 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: S 3939022575:3939022575(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 147282318 0>
17:39:34.872803 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 2590499300 win 8326 <nop,nop,timestamp 147282318 3248254533>
17:39:35.002882 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 41 win 8326 <nop,nop,timestamp 147282331 3248254533>
17:39:37.571659 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: P 0:6(6) ack 41 win 8326 <nop,nop,timestamp 147282585 3248254533>
17:39:37.572923 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: . ack 61 win 8323 <nop,nop,timestamp 147282585 3248254593>
17:39:37.572945 IP brain.cc.rsu.ru.49293 > stat.r61.net.2222: F 6:6(0) ack 61 win 8326 <nop,nop,timestamp 147282585 3248254593>
And firewall counters:
r61net-fbsdhost-1, / # ipfw show
00001 6 326 setfib 1 ip from any to any in recv em1
00010 0 0 count ip from any to any dst-port 2222 fib 0
00011 5 327 count ip from any 2222 to any fib 0
00012 6 326 count ip from any to any dst-port 2222 fib 1
00013 0 0 count ip from any 2222 to any fib 1
00100 0 0 allow ip from any to any via lo0
00200 0 0 deny ip from any to 127.0.0.0/8
00300 0 0 deny ip from 127.0.0.0/8 to any
65000 60 5057 allow ip from any to any
65535 0 0 deny ip from any to any
So the packets, generated by sshd are being sent out via em0 instead of
em1.
With
ipfw add 2 setfib 1 ip from 195.208.245.229 to any
outgoing packets are being tagged with correct fib, but still sent via
em0.
With
ipfw add 60003 fwd 195.208.245.225 src-ip me src-ip 195.208.245.224/27 not dst-ip 195.208.245.224/27
first SYN packet from 195.208.245.229 is being sent correctly via em1,
but I cannot see any further packets at all:
r61net-fbsdhost-1, / # tcpdump -i em1 port 2222
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em1, link-type EN10MB (Ethernet), capture size 96 bytes
18:01:56.665341 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: S 2484180116:2484180116(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 147415433 0>
18:01:56.665463 IP stat.r61.net.2222 > brain.cc.rsu.ru.50435: S 3905497961:3905497961(0) ack 2484180117 win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 3376909218 147415433>
18:01:56.665798 IP brain.cc.rsu.ru.50435 > stat.r61.net.2222: . ack 1 win 8326 <nop,nop,timestamp 147415433 3376909218>
and no packets at em0. TCP connection establishes but no data packets
come from daemon which is rather weird.
I would appreciate any help with this issue.
--
Oleg Sharoyko.
Software and Network Engineer
Computer Center of Rostov State University.
More information about the freebsd-net
mailing list