vnet jail with ipfw having logging problem

[Ian Smith]

On Thu, 2 May 2013 12:09:08 +0200, Anders Hagman wrote:
 > Hi
Yo
 > 2 maj 2013 kl. 07:42 skrev Ian Smith <smithi at nimnet.asn.au>:
 > 
 > > On Wed, 1 May 2013 17:43:03 -0400, Joe wrote:
 > >>>> I have ipfw running inside of a vnet jail on a 9.1-RELEASE host using
 > >>> the
 > >>>> jail(8) definition statements for starting and stopping the vnet jail.
 > >>> As a
 > >>>> side note non-vnet jails are working as expected.
 > >>>>> The host is running a custom kernel with modules and with
 > >>>> options VIMAGE
 > >>>> nooptions SCTP
 > >>>> options IPFIREWALL
 > >>>> options IPFIREWALL_VERBOSE
 > >>>> options IPFIREWALL_VERBOSE_LIMIT=10
 > > 
 > > Please maintain attributions for the archives.  I wrote:
 > > 
 > >>> What steps have you taken during testing to override this ridiculously low
 > >>> limit on logging?  Otherwise, after e.g. just 5 pings and 5 ping responses
 > >>> are logged, all logging ceases until issuing 'ipfw resetlog'.
 > >> 
 > >> /usr/src/sys/conf/NOTES says IPFIREWALL_VERBOSE_LIMIT; limits the number of
 > >> times a matching entry can be logged. Says nothing about this limit being the
 > >> maximum number of log records allowed after which the log file is closed for
 > >> business. Are you saying the /usr/src/sys/conf/NOTES info is no longer true?
 > > 
 > > You showed one (1) 'log' rule for each of the host's and jail's ruleset. 
 > > Once that one rule has been logged 'logamount' times (default as per 
 > > NOTES is 100, but in your case is 10) then logging for THAT rule stops, 
 > > therefore with only one 'log' rule, ALL logging stops.  Understand?
 > > 
 > > If you take the time to properly study the correct reference, ipfw(8), 
 > > all of this will become clear.  See especially section SYSCTL VARIABLES, 
 > > and read thoroughly 'log [logamount number]', at the very least.  Ignore 
 > > the Handbook section on ipfw, it's full of errors and misunderstandings.
 > > 
 > >> Without IPFIREWALL_VERBOSE and IPFIREWALL_VERBOSE_LIMIT where does the logged
 > >> packets get written to? /var/log/security
 > > 
 > > See above.  Both of these options merely set defaults for the sysctls.
 > > 
 > >> I have not used ipfw since it's ipfw2 rewrite so my knowledge is dated.
 > > 
 > > Indeed it is; that's a very long time ago.
 > > 
 > >>>> options IPFIREWALL_DEFAULT_TO_ACCEPT
 > >>>> options IPFIREWALL_IPDIVERT
 > >>> 
 > >>> You'd likely do better using in-kernel NAT; natd doesn't get much love.
 > >>> 
 > >> 
 > >> I kept getting kernel compile errors using "options IPFIREWALL_NAT". I
 > >> thought the error was caused by vimage. Now I know "options LIBALIAS" is
 > >> required. Could not find info on internet search for IPFIREWALL_NAT with
 > >> vimage kernel.
 > > 
 > > Apart from FIREWALL_FORWARD (not even that in 10.x), none of that needs 
 > > to be in the kernel, it's all loadable as modules; see /etc/rc.d/ipfw.
 > > 
 > > If you're doing NAT in the vimage jail, you must have at least two 
 > > interfaces assigned to the jail.  Care to show your config for that?
 > > 
 > >> Do you have first hand experience getting "ipfw kernel nat" to work in a
 > >> vimage jail or having logging work on the host and within the vnet jail?
 > > 
 > > No, but I have just on 15 years experience managing ipfw firewalls :)
 > 
 > When you are new at things you do mistakes, remember.

I still do mistakes.  Trying to teach fishing rather than just tossing 
another fish is often one of mine :)  I'm glad you had some to spare.

 > To try to answer Joes question:
 > 
 > You don't need to compile anything into the kernel regarding ipfw.
 > 
 > Just load the ipfw module in the host system with:
 > 
 >   kldload ipfw
 > 
 > By default a deny all rule is added, so add a allow rule to the host system.
 > 
 >   ipfw add 10 allow ip from any to any
 > 
 > To log things you change the sysctl value net.inet.ip.fw.verbose to 1
 > 
 >   sysctl net.inet.ip.fw.verbose=1
 >
 > If you keep net.inet.ip.fw.verbose_limit=0 you don't have a log limit, and for tests thats fine.

Sure, though the default of 100 is plenty for such tests; it's 
surprisingly easy to DoS syslogd with e.g. a logged flood ping ..

 > log in to the jail system. Change the  sysctl value net.inet.ip.fw.verbose to 1
 > 
 >   sysctl net.inet.ip.fw.verbose=1
 > 
 > Add a logging firewall rule
 > 
 >   ipfw add 10 allow log ip from any to any
 > 
 > Do a ping to an external system.
 > Look inside /var/log/security in the jail system and its empty.

But it does exist, rw for root, with 0 or more bytes, right? And does 
the vimage jail's /etc/syslog.conf contain:
security.*	/var/log/security

That is, I'm checking that the jail's syslogd should be handling these.  
What happens if you run in the jail, say:
 # logger -p security.info Syslog, wherefore art thou, Syslog?
Does that go to the jail's /var/log/security? or the host's?

 > Go to the main host and look at the /var/log/security file and you will find log entries.

Showing the host's hostname, or the jail's?  Can you post some examples?

 > I can confirm Joes bug. I don't have a log rule in the main host but still get log messages.
 > All log messages are from the log rule in the jail system.
 > 
 > System used: 9.1-RELEASE-p2
 > 
 > BR
 > /Anders

Ok, before determining that this is an ipfw-only issue - in which case 
we need to move it over to freebsd-ipfw@ - can you confirm that normal 
syslogging in the jail to /var/log/messages and such is working?

In particular I'm wondering what happens when you do set (say) 
net.inet.ip.fw.verbose_limit=10 and then ping from the jail until 
logging stops .. you should then see a message such as

Apr 23 23:42:05 sola kernel: ipfw: limit 500 reached on entry 26400

both in /var/log/security and in /var/log/messages since it's logged 
as security.notice and default syslog.conf is for *.notice to log to
/var/log/messages .. see the tail of /sys/netpfil/ipfw/ip_fw_log.c

Yes sure, I'm flying blind, don't have a system with jails here yet, and 
am making assumptions about how syslogd(8) should work in jails that I 
really don't have time to properly research currently, nor am I properly
across all the security implications of (particularly vimage) jails.

cheers, Ian