kernel statistics weirdness on RELENG_6{_1,_2} / 64bit?

[Joerg Pernfuss]

Hi,

a while ago I noticed something odd on my 64bit FreeBSD 6 systems - two
amd64, one alpha. I can't make head or tails of it, so I post it in hope
that someone can.

Box #1:
elessar at snake: ~% uname -srm
	FreeBSD 6.1-RELEASE-p7 amd64
elessar at snake: ~% grep kern.ipc /boot/loader.conf
	kern.ipc.nsfbufs="8192"
	kern.ipc.nmbclusters="49152"
elessar at snake: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
	kern.ipc.nsfbufs: 0
	kern.ipc.nmbclusters: 32768
elessar at snake: ~% netstat -m
	350/132/482/49152 mbuf clusters in use (current/cache/total/max)
	0/0/0 sfbufs in use (current/peak/max)
	0 requests for sfbufs denied
	0 requests for sfbufs delayed
	2864770 requests for I/O initiated by sendfile

Box #2:
elessar at forseti: ~% uname -srm
	FreeBSD 6.2-RC1 alpha
elessar at forseti: ~% grep kern.ipc /boot/loader.conf
	kern.ipc.nsfbufs="4294967300"
elessar at forseti: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
	kern.ipc.nsfbufs: 0
	kern.ipc.nmbclusters: 1664
elessar at forseti: ~% netstat -m
	64/80/144/1664 mbuf clusters in use (current/cache/total/max)
	0/0/0 sfbufs in use (current/peak/max)
	0 requests for sfbufs denied
	0 requests for sfbufs delayed
	0 requests for I/O initiated by sendfile

Box #3:
elessar at rivendell: ~% uname -srm
	FreeBSD 6.1-RELEASE-p10 amd64
elessar at rivendell: ~% grep kern.ipc /boot/loader.conf
	kern.ipc.nsfbufs="8192"
	kern.ipc.nmbclusters="49152"
elessar at rivendell: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
	kern.ipc.nsfbufs: 0
	kern.ipc.nmbclusters: 32768
elessar at rivendell: ~% grep NSF /root/kernel/RIVENDELL
	options         NSFBUFS=8192
elessar at rivendell: ~% netstat -m
	64/118/182/49152 mbuf clusters in use (current/cache/total/max)
	0/0/0 sfbufs in use (current/peak/max)
	0 requests for sfbufs denied
	0 requests for sfbufs delayed
	35 requests for I/O initiated by sendfile

For comparison, Box #4:
elessar at web2: ~% uname -srm
	FreeBSD 6.1-STABLE i386
elessar at web2: ~% grep kern.ipc /boot/loader.conf
	kern.ipc.nsfbufs="8192"
elessar at web2: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
	kern.ipc.nsfbufs: 8192
	kern.ipc.nmbclusters: 32768
elessar at web2: ~% netstat -m
	104/272/376/32768 mbuf clusters in use (current/cache/total/max)
	1598/2635/8192 sfbufs in use (current/peak/max)
	0 requests for sfbufs denied
	0 requests for sfbufs delayed
	8938512 requests for I/O initiated by sendfile

So, looking at the `netstat -m' output, setting kern.ipc.nmbclusters
in the loader.conf seems to work, but the sysctl reports the default
value that would be in place, if it wasn't overridden in the loader.conf.

For the sendfile buffers, the stats are completely out of whack.
The sysctl reports 0 for every value (`netstat-m' just prints the sysctl
fields for these values if I understood the source right (unlikely).
But sendfile() works and the accounting on the number of sendfile calls
seems to work too.
So I believe the NSFBUFS kernel option and the loader.conf setting work,
but it is quite impossible to properly tune the buffers to the workload.
I did not bother trying the NMBCLUSTERS kernel option, it was removed
a while ago anyway.

The loader.conf kern.ipc.nsfbufs value on box #2 is ridiculously high,
yes, I tried to get beyond the boundary of a u_int_32 to see if maybe
something good or bad happens. It did not.

At this point, I wanted to include links to two mails on amd64@ I
found when I first searched for this oddity, both of which had no
reply. Instead I found a posting to net@ that occured in the meantime,
and a reply to it by Yar Tikhiy two days ago explaining the nsfbufs issue.
There are no such buffers on amd64, ia64 and alpha as they aren't needed.

http://docs.freebsd.org/cgi/getmsg.cgi?fetch=26091+0+current/freebsd-net

Ok, that leaves the question why `sysctl kern.ipc.nmbclusters' reports a
different value compared to the `netstat -m' output on box #1 and #3.

Thanks for any pointers,
	Joerg

PS: documenting the nsfbufs issue on amd64/ia64/alpha might be a good
PS: idea. loader(8) seems a good place. Opinions?
-- 
| /"\   ASCII ribbon   |  GnuPG Key ID | e86d b753 3deb e749 6c3a |
| \ / campaign against |    0xbbcaad24 | 5706 1f7d 6cfd bbca ad24 |
|  X    HTML in email  |        .the next sentence is true.       |
| / \     and news     |     .the previous sentence was a lie.    |
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20061128/3937a165/signature.pgp