vnet jail with ipfw having logging problem (fwd)

[Ian Smith]

I'm forwarding this from freebsd-jail as it's clearly an ipfw issue.  
Sorry, been totally tied up with $otherstuff since then.

The below conversation is a bit drawn out and in my case, kinda tetchy, 
but the bottom line is that Anders here confirms a bug reported by Joe 
Barbish, that ipfw in a vimage jail is logging its messages to the 
host's /var/log/security and /var/log/messages and not the jail's, 
although other things that log (including logger(1)) do so correctly.

As mentioned, I hunted the logging down to /sys/netpfil/ipfw/ip_fw_log.c 
on 9.1, but in the time available haven't found where log() was defined.
Am I right assuming something's missed being VNET-ed here somewhere?

cheers, Ian

---------- Forwarded message ----------
Date: Thu, 2 May 2013 22:05:49 +0200
From: Anders Hagman <anders.hagman at netplex.se>
To: Ian Smith <smithi at nimnet.asn.au>
Cc: freebsd-jail <freebsd-jail at freebsd.org>
Subject: Re: vnet jail with ipfw having logging problem


2 maj 2013 kl. 18:46 skrev Ian Smith <smithi at nimnet.asn.au>:

> On Thu, 2 May 2013 12:09:08 +0200, Anders Hagman wrote:
>> Hi
> Yo
>> 2 maj 2013 kl. 07:42 skrev Ian Smith <smithi at nimnet.asn.au>:
>> 
>>> On Wed, 1 May 2013 17:43:03 -0400, Joe wrote:
>>>>>> I have ipfw running inside of a vnet jail on a 9.1-RELEASE host using
>>>>> the
>>>>>> jail(8) definition statements for starting and stopping the vnet jail.
>>>>> As a
>>>>>> side note non-vnet jails are working as expected.
>>>>>>> The host is running a custom kernel with modules and with
>>>>>> options VIMAGE
>>>>>> nooptions SCTP
>>>>>> options IPFIREWALL
>>>>>> options IPFIREWALL_VERBOSE
>>>>>> options IPFIREWALL_VERBOSE_LIMIT=10
>>> 
>>> Please maintain attributions for the archives.  I wrote:
>>> 
>>>>> What steps have you taken during testing to override this ridiculously low
>>>>> limit on logging?  Otherwise, after e.g. just 5 pings and 5 ping responses
>>>>> are logged, all logging ceases until issuing 'ipfw resetlog'.
>>>> 
>>>> /usr/src/sys/conf/NOTES says IPFIREWALL_VERBOSE_LIMIT; limits the number of
>>>> times a matching entry can be logged. Says nothing about this limit being the
>>>> maximum number of log records allowed after which the log file is closed for
>>>> business. Are you saying the /usr/src/sys/conf/NOTES info is no longer true?
>>> 
>>> You showed one (1) 'log' rule for each of the host's and jail's ruleset. 
>>> Once that one rule has been logged 'logamount' times (default as per 
>>> NOTES is 100, but in your case is 10) then logging for THAT rule stops, 
>>> therefore with only one 'log' rule, ALL logging stops.  Understand?
>>> 
>>> If you take the time to properly study the correct reference, ipfw(8), 
>>> all of this will become clear.  See especially section SYSCTL VARIABLES, 
>>> and read thoroughly 'log [logamount number]', at the very least.  Ignore 
>>> the Handbook section on ipfw, it's full of errors and misunderstandings.
>>> 
>>>> Without IPFIREWALL_VERBOSE and IPFIREWALL_VERBOSE_LIMIT where does the logged
>>>> packets get written to? /var/log/security
>>> 
>>> See above.  Both of these options merely set defaults for the sysctls.
>>> 
>>>> I have not used ipfw since it's ipfw2 rewrite so my knowledge is dated.
>>> 
>>> Indeed it is; that's a very long time ago.
>>> 
>>>>>> options IPFIREWALL_DEFAULT_TO_ACCEPT
>>>>>> options IPFIREWALL_IPDIVERT
>>>>> 
>>>>> You'd likely do better using in-kernel NAT; natd doesn't get much love.
>>>>> 
>>>> 
>>>> I kept getting kernel compile errors using "options IPFIREWALL_NAT". I
>>>> thought the error was caused by vimage. Now I know "options LIBALIAS" is
>>>> required. Could not find info on internet search for IPFIREWALL_NAT with
>>>> vimage kernel.
>>> 
>>> Apart from FIREWALL_FORWARD (not even that in 10.x), none of that needs 
>>> to be in the kernel, it's all loadable as modules; see /etc/rc.d/ipfw.
>>> 
>>> If you're doing NAT in the vimage jail, you must have at least two 
>>> interfaces assigned to the jail.  Care to show your config for that?
>>> 
>>>> Do you have first hand experience getting "ipfw kernel nat" to work in a
>>>> vimage jail or having logging work on the host and within the vnet jail?
>>> 
>>> No, but I have just on 15 years experience managing ipfw firewalls :)
>> 
>> When you are new at things you do mistakes, remember.
> 
> I still do mistakes.  Trying to teach fishing rather than just tossing 
> another fish is often one of mine :)  I'm glad you had some to spare.

I know the game. ;->

> 
>> To try to answer Joes question:
>> 
>> You don't need to compile anything into the kernel regarding ipfw.
>> 
>> Just load the ipfw module in the host system with:
>> 
>>  kldload ipfw
>> 
>> By default a deny all rule is added, so add a allow rule to the host system.
>> 
>>  ipfw add 10 allow ip from any to any
>> 
>> To log things you change the sysctl value net.inet.ip.fw.verbose to 1
>> 
>>  sysctl net.inet.ip.fw.verbose=1
>> 
>> If you keep net.inet.ip.fw.verbose_limit=0 you don't have a log limit, and for tests thats fine.
> 
> Sure, though the default of 100 is plenty for such tests; it's 
> surprisingly easy to DoS syslogd with e.g. a logged flood ping ..
> 
>> log in to the jail system. Change the  sysctl value net.inet.ip.fw.verbose to 1
>> 
>>  sysctl net.inet.ip.fw.verbose=1
>> 
>> Add a logging firewall rule
>> 
>>  ipfw add 10 allow log ip from any to any
>> 
>> Do a ping to an external system.
>> Look inside /var/log/security in the jail system and its empty.
> 
> But it does exist, rw for root, with 0 or more bytes, right? And does 
> the vimage jail's /etc/syslog.conf contain:
> security.*	/var/log/security
> 

Yes

> That is, I'm checking that the jail's syslogd should be handling these.  
> What happens if you run in the jail, say:
> # logger -p security.info Syslog, wherefore art thou, Syslog?
> Does that go to the jail's /var/log/security? or the host's?

In jail system webben:
logger -p security.info Syslog, wherefore art thou, Syslog?

tail /var/log/security
May  2 21:24:48 webben root: Syslog, wherefore art thou, Syslog?

> 
>> Go to the main host and look at the /var/log/security file and you will find log entries.
> 
> Showing the host's hostname, or the jail's?  Can you post some examples?

In host system dator5:
tail /var/log/security
May  2 21:29:15 dator5 kernel: ipfw: 10 Accept TCP 10.2.0.101:80 94.153.64.32:3085 out via vlan101
May  2 21:29:15 dator5 kernel: ipfw: 10 Accept TCP 94.153.64.32:3085 10.2.0.101:80 in via vlan101

> 
>> I can confirm Joes bug. I don't have a log rule in the main host but still get log messages.
>> All log messages are from the log rule in the jail system.
>> 
>> System used: 9.1-RELEASE-p2
>> 
>> BR
>> /Anders
> 
> Ok, before determining that this is an ipfw-only issue - in which case 
> we need to move it over to freebsd-ipfw@ - can you confirm that normal 
> syslogging in the jail to /var/log/messages and such is working?
> 
In jail system
login anders
password *****

tail /var/log/messages
May  2 21:41:57 webben login: login_getclass: unknown class 'svensk'
May  2 21:42:00 webben last message repeated 3 times

> In particular I'm wondering what happens when you do set (say) 
> net.inet.ip.fw.verbose_limit=10 and then ping from the jail until 
> logging stops .. you should then see a message such as
> 
> Apr 23 23:42:05 sola kernel: ipfw: limit 500 reached on entry 26400
> 
> both in /var/log/security and in /var/log/messages since it's logged 
> as security.notice and default syslog.conf is for *.notice to log to
> /var/log/messages .. see the tail of /sys/netpfil/ipfw/ip_fw_log.c
> 
> Yes sure, I'm flying blind, don't have a system with jails here yet, and 
> am making assumptions about how syslogd(8) should work in jails that I 
> really don't have time to properly research currently, nor am I properly
> across all the security implications of (particularly vimage) jails.
> 

On jail system:
  
sysctl net.inet.ip.fw.verbose_limit=10

Pinging repeatedly. Just continue to log to host system.

Add new ipfw log role will use the new limit:

 ipfw add 5 allow log ip from any to any
 00005 allow log logamount 10 ip from any to any

New ping test.
/var/log/security in host system :

May  2 21:52:28 dator5 kernel: ipfw: 5 Accept ICMP:8.0 10.2.0.101 195.49.241.132 out via vlan101
May  2 21:52:28 dator5 kernel: ipfw: 5 Accept ICMP:0.0 195.49.241.132 10.2.0.101 in via vlan101
May  2 21:52:28 dator5 kernel: ipfw: limit 10 reached on entry 5

/var/log/messages in host system :

May  2 21:52:28 dator5 kernel: ipfw: limit 10 reached on entry 5

Nothing at all is logged to the jail syslog.

BR
/Anders