kernel statistics weirdness on RELENG_6{_1,_2} / 64bit?
Yar Tikhiy
yar at comp.chem.msu.su
Tue Nov 28 23:33:58 PST 2006
On Tue, Nov 28, 2006 at 07:54:52PM +0100, Joerg Pernfuss wrote:
> Hi,
>
> a while ago I noticed something odd on my 64bit FreeBSD 6 systems - two
> amd64, one alpha. I can't make head or tails of it, so I post it in hope
> that someone can.
>
> Box #1:
> elessar at snake: ~% uname -srm
> FreeBSD 6.1-RELEASE-p7 amd64
> elessar at snake: ~% grep kern.ipc /boot/loader.conf
> kern.ipc.nsfbufs="8192"
> kern.ipc.nmbclusters="49152"
> elessar at snake: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
> kern.ipc.nsfbufs: 0
> kern.ipc.nmbclusters: 32768
> elessar at snake: ~% netstat -m
> 350/132/482/49152 mbuf clusters in use (current/cache/total/max)
> 0/0/0 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 2864770 requests for I/O initiated by sendfile
>
> Box #2:
> elessar at forseti: ~% uname -srm
> FreeBSD 6.2-RC1 alpha
> elessar at forseti: ~% grep kern.ipc /boot/loader.conf
> kern.ipc.nsfbufs="4294967300"
> elessar at forseti: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
> kern.ipc.nsfbufs: 0
> kern.ipc.nmbclusters: 1664
> elessar at forseti: ~% netstat -m
> 64/80/144/1664 mbuf clusters in use (current/cache/total/max)
> 0/0/0 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
>
> Box #3:
> elessar at rivendell: ~% uname -srm
> FreeBSD 6.1-RELEASE-p10 amd64
> elessar at rivendell: ~% grep kern.ipc /boot/loader.conf
> kern.ipc.nsfbufs="8192"
> kern.ipc.nmbclusters="49152"
> elessar at rivendell: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
> kern.ipc.nsfbufs: 0
> kern.ipc.nmbclusters: 32768
> elessar at rivendell: ~% grep NSF /root/kernel/RIVENDELL
> options NSFBUFS=8192
> elessar at rivendell: ~% netstat -m
> 64/118/182/49152 mbuf clusters in use (current/cache/total/max)
> 0/0/0 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 35 requests for I/O initiated by sendfile
>
> For comparison, Box #4:
> elessar at web2: ~% uname -srm
> FreeBSD 6.1-STABLE i386
> elessar at web2: ~% grep kern.ipc /boot/loader.conf
> kern.ipc.nsfbufs="8192"
> elessar at web2: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters
> kern.ipc.nsfbufs: 8192
> kern.ipc.nmbclusters: 32768
> elessar at web2: ~% netstat -m
> 104/272/376/32768 mbuf clusters in use (current/cache/total/max)
> 1598/2635/8192 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 8938512 requests for I/O initiated by sendfile
>
> So, looking at the `netstat -m' output, setting kern.ipc.nmbclusters
> in the loader.conf seems to work, but the sysctl reports the default
> value that would be in place, if it wasn't overridden in the loader.conf.
>
> For the sendfile buffers, the stats are completely out of whack.
> The sysctl reports 0 for every value (`netstat-m' just prints the sysctl
> fields for these values if I understood the source right (unlikely).
> But sendfile() works and the accounting on the number of sendfile calls
> seems to work too.
> So I believe the NSFBUFS kernel option and the loader.conf setting work,
> but it is quite impossible to properly tune the buffers to the workload.
> I did not bother trying the NMBCLUSTERS kernel option, it was removed
> a while ago anyway.
>
> The loader.conf kern.ipc.nsfbufs value on box #2 is ridiculously high,
> yes, I tried to get beyond the boundary of a u_int_32 to see if maybe
> something good or bad happens. It did not.
>
> At this point, I wanted to include links to two mails on amd64@ I
> found when I first searched for this oddity, both of which had no
> reply. Instead I found a posting to net@ that occured in the meantime,
> and a reply to it by Yar Tikhiy two days ago explaining the nsfbufs issue.
> There are no such buffers on amd64, ia64 and alpha as they aren't needed.
>
> http://docs.freebsd.org/cgi/getmsg.cgi?fetch=26091+0+current/freebsd-net
>
> Ok, that leaves the question why `sysctl kern.ipc.nmbclusters' reports a
> different value compared to the `netstat -m' output on box #1 and #3.
netstat -m doesn't use the sysctl. It reads the zone limit via
memstat_get_countlimit() from libmemstat(3) instead. Perhaps this
can account for the difference observed.
> Thanks for any pointers,
> Joerg
>
> PS: documenting the nsfbufs issue on amd64/ia64/alpha might be a good
> PS: idea. loader(8) seems a good place. Opinions?
I've already documented the issue in sendfile(2) in CURRENT. MFC
is due. A small note in loader(8) would be good, too -- thanks for
the suggestion.
--
Yar
More information about the freebsd-stable
mailing list