jail source address selection doesn't work?

[Alex Povolotsky]

03.03.2011 0:48, Bjoern A. Zeeb пишет:
> On Mon, 7 Feb 2011, Alex Povolotsky wrote:
>
>> Hello!
>>
>> On a multihomed FreeBSD 8.1-RELEASE, in a multihomed jail, source IP 
>> selection suddenly refused to work.
>>
>> ifconfig on a box:
> ....
>> Seems reasonable, yes?
>>
>> Pinging from the box
>>
>> # ping 192.168.75.59
>> PING 192.168.75.59 (192.168.75.59): 56 data bytes
>> 64 bytes from 192.168.75.59: icmp_seq=0 ttl=64 time=0.993 ms
>> 64 bytes from 192.168.75.59: icmp_seq=1 ttl=64 time=0.986 ms
>> 64 bytes from 192.168.75.59: icmp_seq=2 ttl=64 time=0.988 ms
>> ^C
>> --- 192.168.75.59 ping statistics ---
>> 3 packets transmitted, 3 packets received, 0.0% packet loss
>> round-trip min/avg/max/stddev = 0.986/0.989/0.993/0.003 ms
>>
>> 10:45:31.425232 IP 192.168.75.4 > 192.168.75.59: ICMP echo request, 
>> id 12430, seq 0, length 64
>> 10:45:31.426283 IP 192.168.75.59 > 192.168.75.4: ICMP echo reply, id 
>> 12430, seq 0, length 64
>> 10:45:32.425415 IP 192.168.75.4 > 192.168.75.59: ICMP echo request, 
>> id 12430, seq 1, length 64
>> 10:45:32.426404 IP 192.168.75.59 > 192.168.75.4: ICMP echo reply, id 
>> 12430, seq 1, length 64
>>
>> Okay, yes?
>>
>> From jail:
>>
>> # ping 192.168.75.59
>> PING 192.168.75.59 (192.168.75.59): 56 data bytes
>> ^C
>> --- 192.168.75.59 ping statistics ---
>> 2 packets transmitted, 0 packets received, 100.0% packet loss
>>
>> 10:45:52.146600 IP 83.69.203.1 > 192.168.75.59: ICMP echo request, id 
>> 14222, seq 0, length 64
>> 10:45:53.146702 IP 83.69.203.1 > 192.168.75.59: ICMP echo request, id 
>> 14222, seq 1, length 64
>>
>> Setting ip.saddrsel to 1 or 0 did not change anything. Kernel is 
>> GENERIC+ALTQ
>>
>> What could I miss?...
>
> Don't use ping to test this. a) for ping inside the jail to work you
> need to enable raw sockets b) a) could give you a hint that ping does
> it's own thing.
Telnet did all the same thing.
>
> Try a telnet to a random port to the destination and verify with
> tcpdump whether things are still not working correctly, of if you
> establish the connection with netstat.
I used telnet to connect to specific ports.

Ok, let's try again

104:tarkhil at box2.u.energodata.local:...local/etc/ezjail # jls
JID IP Address Hostname Path
1 192.168.82.2 test /usr/jails/test
107:tarkhil at box2.u.energodata.local:...local/etc/ezjail # jls -j 1 
ip4.saddrsel
true
108:tarkhil at box2.u.energodata.local:...local/etc/ezjail # jls -j 1 ip4.addr
192.168.82.2,192.168.75.2
114:tarkhil at box2.u.energodata.local:...local/etc/ezjail # tcpdump -l -n 
-i bce0 host 192.168.82.2
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bce0, link-type EN10MB (Ethernet), capture size 96 bytes
09:27:54.492105 IP 192.168.82.2.50823 > 192.168.72.3.22: Flags [S], seq 
3819433473, win 65535, options [mss 1460,nop,wscale 3,sackOK,TS val 
1306232522 ecr 0], length 0
115:tarkhil at box2.u.energodata.local:...local/etc/ezjail # ifconfig bce0
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 00:14:5e:1a:a6:27
inet 192.168.80.41 netmask 0xffffff00 broadcast 192.168.80.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
test# sysctl security.jail.jailed
security.jail.jailed: 1
test# ifconfig
bce0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 00:14:5e:1a:a6:27
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c01bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether 00:14:5e:1a:a6:29
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=3<RXCSUM,TXCSUM>
vlan75: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:14:5e:1a:a6:29
inet 192.168.75.2 netmask 0xffffff00 broadcast 192.168.75.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 75 parent interface: bce1
vlan82: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=103<RXCSUM,TXCSUM,TSO4>
ether 00:14:5e:1a:a6:29
inet 192.168.82.2 netmask 0xffffff00 broadcast 192.168.82.255
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
vlan: 82 parent interface: bce1

In other words, source address is selected as primary IP, and packet 
runs out on 100% improper interface.

No specific routing, no firewall.

Alex.