vnet jail with ipfw having logging problem
Joe
fbsd8 at a1poweruser.com
Fri May 3 00:54:37 UTC 2013
I am posting 2 console logs created using the script command.
The main differences between the 2 is,
log 1 is a 9.1 kernel with modules and vimage compiled in. This shows
the first problem being that dynamically loaded ipfw with a vimage
kernel don't work.
Log 2 is a 9.1 kernel with modules and vimage plus ipfw compiled in.
This shows the second problem with vnet jails running ipfw log to host
security file and don't log any ipfw log messages to the hosts message
file. Secondly the vnet jails security and messages files never get
populated with ipfw log messages.
Console log 1.
9.1-RELEASE ipfw dynamically loaded by firewall statements in hosts
rc.conf with modules and only vimage compiled into kernel.
logger cmd on host did not work until after vnet jail was started and
stopped.
vnet jail pings passed through vnet jail but was not handed to host ipfw.
vnet jail pings got logged to hosts security file but not messages.
After vnet jail stopped, host logger cmd works and host pings work and
logged correctly to security and messages.
# /root >sysctl net.inet.ip.fw.verbose
net.inet.ip.fw.verbose: 1
# /root >sysctl net.inet.ip.fw.verbose_limit
net.inet.ip.fw.verbose_limit: 0
# /root >cat /etc/rc.comf
#
snip
firewall_enable="YES"
firewall_logging="YES"
firewall_script="/etc/ipfw.rules"
# /root >logger security.notice this msg is from logger cmd on host
# /root >cat /var/log/security
empty file
# /root >cat /var/log/messages
empty file
# /root >ping -c 4 freebsd.org
PING freebsd.org (8.8.178.135): 56 data bytes
64 bytes from 8.8.178.135: icmp_seq=0 ttl=51 time=102.814 ms
64 bytes from 8.8.178.135: icmp_seq=1 ttl=51 time=84.625 ms
64 bytes from 8.8.178.135: icmp_seq=2 ttl=51 time=101.332 ms
64 bytes from 8.8.178.135: icmp_seq=3 ttl=51 time=120.662 ms
--- freebsd.org ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 84.625/102.358/120.662/12.755 ms
# /root >cat /var/log/messages
empty file
# /root >cat /var/log/security
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.5:42524
209.18.47.61:53 out via rl0
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53
10.0.10.5:42524 in via rl0
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
# /root >logger security.notice this msg is from logger cmd on host
# /root >cat /var/log/security
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.5:42524
209.18.47.61:53 out via rl0
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53
10.0.10.5:42524 in via rl0
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
vnet jail gets started
# /root >jls
JID IP Address Hostname Path
2 - vdir2 /usr/jails/vdir2
# /root >jexec vdir2 tcsh
vdir2 / >logger -p security.notice logger cmd msg from within the host
vdir2 / >ipfw -a list
00010 0 0 allow ip from any to any via lo0
00011 0 0 allow log ip from any to any via epair2b
65535 5 368 deny ip from any to any
vdir2 / >ping -c 4 freebsd.org
ping: cannot resolve freebsd.org: Host name lookup failure
vdir2 / >ipfw -a list
00010 0 0 allow ip from any to any via lo0
00011 8 480 allow log ip from any to any via epair2b
65535 5 368 deny ip from any to any
vdir2 / >exit
exit
# back on the host
# /root >cat /var/log/security
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.5:42524
209.18.47.61:53 out via rl0
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53
10.0.10.5:42524 in via rl0
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:50 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:51 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:52 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:05:53 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:10:50 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:32606
209.18.47.61:53 out via epair2b
May 2 19:10:55 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:29810
209.18.47.62:53 out via epair2b
May 2 19:10:57 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:32606
209.18.47.61:53 out via epair2b
May 2 19:11:00 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:35933
209.18.47.61:53 out via epair2b
May 2 19:11:05 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:56823
209.18.47.62:53 out via epair2b
May 2 19:11:07 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:35933
209.18.47.61:53 out via epair2b
May 2 19:11:07 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:29810
209.18.47.62:53 out via epair2b
May 2 19:11:17 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:56823
209.18.47.62:53 out via epair2b
May 2 19:11:22 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:37981
209.18.47.61:53 out via epair2b
May 2 19:11:27 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:24567
209.18.47.62:53 out via epair2b
May 2 19:11:29 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:37981
209.18.47.61:53 out via epair2b
May 2 19:11:39 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:24567
209.18.47.62:53 out via epair2b
May 2 19:11:44 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:54854
209.18.47.61:53 out via epair2b
May 2 19:11:49 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:33964
209.18.47.62:53 out via epair2b
May 2 19:11:51 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:54854
209.18.47.61:53 out via epair2b
# /root >logger -p security.notice host logger msg
# /root >cat /var/log/security
May 2 19:11:39 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:24567
209.18.47.62:53 out via epair2b
May 2 19:11:44 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:54854
209.18.47.61:53 out via epair2b
May 2 19:11:49 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:33964
209.18.47.62:53 out via epair2b
May 2 19:11:51 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:54854
209.18.47.61:53 out via epair2b
May 2 19:12:01 fbsdjones kernel: ipfw: 11 Accept UDP 10.2.0.2:33964
209.18.47.62:53 out via epair2b
May 2 19:12:50 fbsdjones root: host logger msg
# /root >cat /var/log/messages
May 2 19:08:10 fbsdjones kernel: bridge0: Ethernet address:
02:8f:94:84:0c:00
May 2 19:08:10 fbsdjones kernel: bridge0: link state changed to UP
May 2 19:08:10 fbsdjones kernel: epair2a: Ethernet address:
02:c0:a4:00:0a:0a
May 2 19:08:10 fbsdjones kernel: epair2b: Ethernet address:
02:c0:a4:00:0b:0b
May 2 19:08:10 fbsdjones kernel: epair2a: link state changed to UP
May 2 19:08:10 fbsdjones kernel: epair2b: link state changed to UP
May 2 19:12:50 fbsdjones root: host logger msg
Console log 2.
This test run is using 9.1-RELEASE with modules plus vimage and ipfw
compiled in.
options IPFIREWALL
options IPFIREWALL_VERBOSE
options IPFIREWALL_DEFAULT_TO_ACCEPT
logger command works. logged msg in both security and messages on host
vnet jail can ping the public internet.
Hosts security file has log messages from both jail and host.
ipfw log messages are not being put into the hosts messages file.
# ran on the host
# /root >sysctl net.inet.ip.fw.verbose
net.inet.ip.fw.verbose: 1
# /root >sysctl net.inet.ip.fw.verbose_limit
net.inet.ip.fw.verbose_limit: 0
# /root >ipfw -a list
00010 0 0 allow ip from any to any via lo0
00011 0 0 allow log ip from any to any via rl0
65535 1 328 allow ip from any to any
# /root >/var/log/security
empty file
# /root >cat /var/log/messages
empty file
# /root >logger -p security.notice host logger cmd 1
# /root >cat /var/log/security
May 2 19:45:51 fbsdjones root: host logger cmd 1
# /root >cat /var/log/messages
May 2 19:45:51 fbsdjones root: host logger cmd 1
# /root >ipfw -a list
00010 0 0 allow ip from any to any via lo0
00011 0 0 allow log ip from any to any via rl0
65535 1 328 allow ip from any to any
# /root >ping -c 3 freebsd.org
PING freebsd.org (8.8.178.135): 56 data bytes
64 bytes from 8.8.178.135: icmp_seq=0 ttl=51 time=85.032 ms
64 bytes from 8.8.178.135: icmp_seq=1 ttl=51 time=84.381 ms
64 bytes from 8.8.178.135: icmp_seq=2 ttl=51 time=84.647 ms
--- freebsd.org ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 84.381/84.687/85.032/0.267 ms
# /root >ipfw -a list
00010 0 0 allow ip from any to any via lo0
00011 9 869 allow log ip from any to any via rl0
65535 1 328 allow ip from any to any
vnet jail started
# /root >jls
JID IP Address Hostname Path
1 - vdir2 /usr/jails/vdir2
# /root >jexec vdir2 tcsh
vdir2 / >cat /etc/ipfw.rules
# Flush out the list before we begin.
ipfw -q -f flush
cmd="ipfw -q add"
if [ -e /etc/epair ]; then
pif=`cat "/etc/epair"`
rm /etc/epair
else
pif="lo0"
fi
$cmd 010 allow all from any to any via lo0
$cmd 011 allow log all from any to any via $pif
vdir2 / >ipfw -a list
00010 0 0 allow ip from any to any via lo0
00011 0 0 allow log ip from any to any via epair1b
65535 8 624 allow ip from any to any
vdir2 / >ping -c 3 freebsd.org
PING freebsd.org (8.8.178.135): 56 data bytes
64 bytes from 8.8.178.135: icmp_seq=0 ttl=51 time=84.342 ms
64 bytes from 8.8.178.135: icmp_seq=1 ttl=51 time=84.195 ms
64 bytes from 8.8.178.135: icmp_seq=2 ttl=51 time=84.015 ms
--- freebsd.org ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 84.015/84.184/84.342/0.134 ms
vdir2 / >ipfw -a list
00010 0 0 allow ip from any to any via lo0
00011 8 634 allow log ip from any to any via epair1b
65535 8 624 allow ip from any to any
vdir2 / >cat /var/log/security
May 1 21:56:27 vdir2 newsyslog[5202]: logfile first created
vdir2 / >cat /var/log/messages
May 1 21:56:27 vdir2 newsyslog[5202]: logfile first created
vdir2 / >exit
exit
Back on the host
# /root >cat /var/log/security
May 2 19:45:51 fbsdjones root: host logger cmd 1
May 2 19:46:53 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.1:138
10.0.10.7:138 in via rl0
May 2 19:46:58 fbsdjones kernel: ipfw: 11 Accept UDP 10.0.10.5:64721
209.18.47.61:53 out via rl0
May 2 19:46:58 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53
10.0.10.5:64721 in via rl0
May 2 19:46:58 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:46:58 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:46:59 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:46:59 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:47:00 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.0.10.5
8.8.178.135 out via rl0
May 2 19:47:00 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.0.10.5 in via rl0
May 2 19:47:38 fbsdjones kernel: ipfw: 11 Accept ICMPv6:143.0 [::]
[ff02::16] out via rl0
May 2 19:47:38 fbsdjones kernel: ipfw: 11 Accept ICMPv6:143.0 [::]
[ff02::16] out via rl0
May 2 19:47:39 fbsdjones kernel: ipfw: 11 Accept ICMPv6:135.0 [::]
[ff02::1:ff00:b0b] out via rl0
May 2 19:47:39 fbsdjones kernel: ipfw: 11 Accept ICMPv6:143.0 [::]
[ff02::16] out via rl0
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 10.1.0.2:13101
209.18.47.61:53 out via epair1b
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 10.1.0.2:13101
209.18.47.61:53 out via rl0
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53
10.1.0.2:13101 in via rl0
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53
10.1.0.2:13101 in via rl0
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept UDP 209.18.47.61:53
10.1.0.2:13101 in via epair1b
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2
8.8.178.135 out via epair1b
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2
8.8.178.135 out via rl0
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.1.0.2 in via rl0
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.1.0.2 in via rl0
May 2 19:49:21 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.1.0.2 in via epair1b
May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2
8.8.178.135 out via epair1b
May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2
8.8.178.135 out via rl0
May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.1.0.2 in via rl0
May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.1.0.2 in via rl0
May 2 19:49:22 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.1.0.2 in via epair1b
May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2
8.8.178.135 out via epair1b
May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:8.0 10.1.0.2
8.8.178.135 out via rl0
May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.1.0.2 in via rl0
May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.1.0.2 in via rl0
May 2 19:49:23 fbsdjones kernel: ipfw: 11 Accept ICMP:0.0 8.8.178.135
10.1.0.2 in via epair1b
# /root >cat /var/log/messages
May 2 19:45:51 fbsdjones root: host logger cmd 1
May 2 19:47:38 fbsdjones kernel: bridge0: Ethernet address:
02:8f:94:84:0c:00
May 2 19:47:38 fbsdjones kernel: bridge0: link state changed to UP
May 2 19:47:38 fbsdjones kernel: epair1a: Ethernet address:
02:c0:24:00:0a:0a
May 2 19:47:38 fbsdjones kernel: epair1b: Ethernet address:
02:c0:24:00:0b:0b
May 2 19:47:38 fbsdjones kernel: epair1a: link state changed to UP
May 2 19:47:38 fbsdjones kernel: epair1b: link state changed to UP
May 2 19:50:59 fbsdjones kernel: epair1a: link state changed to DOWN
May 2 19:50:59 fbsdjones kernel: epair1b: link state changed to DOWN
May 2 19:50:59 fbsdjones kernel: bridge0: link state changed to DOWN
May 2 19:51:02 fbsdjones kernel: Freed UMA keg was not empty (30
items). Lost 2 pages of memory.
May 2 19:51:02 fbsdjones kernel: Freed UMA keg was not empty (203
items). Lost 1 pages of memory.
May 2 19:51:02 fbsdjones kernel: Freed UMA keg was not empty (30
items). Lost 2 pages of memory.
May 2 19:51:02 fbsdjones kernel: hhook_vnet_uninit: hhook_head type=1,
id=1 cleanup required
May 2 19:51:02 fbsdjones kernel: hhook_vnet_uninit: hhook_head type=1,
id=0 cleanup required
# /root >exit
exit
More information about the freebsd-jail
mailing list