kernel statistics weirdness on RELENG_6{_1,_2} / 64bit?

[Yar Tikhiy]

On Tue, Nov 28, 2006 at 07:54:52PM +0100, Joerg Pernfuss wrote:
> Hi,
> 
> a while ago I noticed something odd on my 64bit FreeBSD 6 systems - two
> amd64, one alpha. I can't make head or tails of it, so I post it in hope
> that someone can.
> 
> Box #1:
> elessar at snake: ~% uname -srm
> 	FreeBSD 6.1-RELEASE-p7 amd64
> elessar at snake: ~% grep kern.ipc /boot/loader.conf
> 	kern.ipc.nsfbufs="8192"
> 	kern.ipc.nmbclusters="49152"
> elessar at snake: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
> 	kern.ipc.nsfbufs: 0
> 	kern.ipc.nmbclusters: 32768
> elessar at snake: ~% netstat -m
> 	350/132/482/49152 mbuf clusters in use (current/cache/total/max)
> 	0/0/0 sfbufs in use (current/peak/max)
> 	0 requests for sfbufs denied
> 	0 requests for sfbufs delayed
> 	2864770 requests for I/O initiated by sendfile
> 
> Box #2:
> elessar at forseti: ~% uname -srm
> 	FreeBSD 6.2-RC1 alpha
> elessar at forseti: ~% grep kern.ipc /boot/loader.conf
> 	kern.ipc.nsfbufs="4294967300"
> elessar at forseti: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
> 	kern.ipc.nsfbufs: 0
> 	kern.ipc.nmbclusters: 1664
> elessar at forseti: ~% netstat -m
> 	64/80/144/1664 mbuf clusters in use (current/cache/total/max)
> 	0/0/0 sfbufs in use (current/peak/max)
> 	0 requests for sfbufs denied
> 	0 requests for sfbufs delayed
> 	0 requests for I/O initiated by sendfile
> 
> Box #3:
> elessar at rivendell: ~% uname -srm
> 	FreeBSD 6.1-RELEASE-p10 amd64
> elessar at rivendell: ~% grep kern.ipc /boot/loader.conf
> 	kern.ipc.nsfbufs="8192"
> 	kern.ipc.nmbclusters="49152"
> elessar at rivendell: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
> 	kern.ipc.nsfbufs: 0
> 	kern.ipc.nmbclusters: 32768
> elessar at rivendell: ~% grep NSF /root/kernel/RIVENDELL
> 	options         NSFBUFS=8192
> elessar at rivendell: ~% netstat -m
> 	64/118/182/49152 mbuf clusters in use (current/cache/total/max)
> 	0/0/0 sfbufs in use (current/peak/max)
> 	0 requests for sfbufs denied
> 	0 requests for sfbufs delayed
> 	35 requests for I/O initiated by sendfile
> 
> For comparison, Box #4:
> elessar at web2: ~% uname -srm
> 	FreeBSD 6.1-STABLE i386
> elessar at web2: ~% grep kern.ipc /boot/loader.conf
> 	kern.ipc.nsfbufs="8192"
> elessar at web2: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
> 	kern.ipc.nsfbufs: 8192
> 	kern.ipc.nmbclusters: 32768
> elessar at web2: ~% netstat -m
> 	104/272/376/32768 mbuf clusters in use (current/cache/total/max)
> 	1598/2635/8192 sfbufs in use (current/peak/max)
> 	0 requests for sfbufs denied
> 	0 requests for sfbufs delayed
> 	8938512 requests for I/O initiated by sendfile
> 
> So, looking at the `netstat -m' output, setting kern.ipc.nmbclusters
> in the loader.conf seems to work, but the sysctl reports the default
> value that would be in place, if it wasn't overridden in the loader.conf.
> 
> For the sendfile buffers, the stats are completely out of whack.
> The sysctl reports 0 for every value (`netstat-m' just prints the sysctl
> fields for these values if I understood the source right (unlikely).
> But sendfile() works and the accounting on the number of sendfile calls
> seems to work too.
> So I believe the NSFBUFS kernel option and the loader.conf setting work,
> but it is quite impossible to properly tune the buffers to the workload.
> I did not bother trying the NMBCLUSTERS kernel option, it was removed
> a while ago anyway.
> 
> The loader.conf kern.ipc.nsfbufs value on box #2 is ridiculously high,
> yes, I tried to get beyond the boundary of a u_int_32 to see if maybe
> something good or bad happens. It did not.
> 
> At this point, I wanted to include links to two mails on amd64@ I
> found when I first searched for this oddity, both of which had no
> reply. Instead I found a posting to net@ that occured in the meantime,
> and a reply to it by Yar Tikhiy two days ago explaining the nsfbufs issue.
> There are no such buffers on amd64, ia64 and alpha as they aren't needed.
> 
> http://docs.freebsd.org/cgi/getmsg.cgi?fetch=26091+0+current/freebsd-net
> 
> Ok, that leaves the question why `sysctl kern.ipc.nmbclusters' reports a
> different value compared to the `netstat -m' output on box #1 and #3.

netstat -m doesn't use the sysctl.  It reads the zone limit via
memstat_get_countlimit() from libmemstat(3) instead.  Perhaps this
can account for the difference observed.

> Thanks for any pointers,
> 	Joerg
> 
> PS: documenting the nsfbufs issue on amd64/ia64/alpha might be a good
> PS: idea. loader(8) seems a good place. Opinions?

I've already documented the issue in sendfile(2) in CURRENT.  MFC
is due.  A small note in loader(8) would be good, too -- thanks for
the suggestion.

-- 
Yar