"Out of mbuf address space!"

[Grant Peel]

Hi all,

I have an older RAID 5 machine running FreeBSD 5.2.1 and am using it as a backup storage unit.

Yesterday morning, we noticed the the NFS mounts on the clients to this machine we not available, which sent a bunch of cronjobs spinning out of control etc. We also became unable to connect via ssh.

Once at the console we noted sevral dozen entries in the messages.log:

Oct  1 08:32:13 enterprise kernel: Out of mbuf address space!
Oct  1 08:32:13 enterprise kernel: Consider increasing NMBCLUSTERS
Oct  1 08:32:13 enterprise kernel: All mbufs or mbuf clusters exhausted, please see tuning(7).

After rebooting the machine, and getting the clients under control I started investigating tunning(7) in the man pages.

I am confused however.

I have increased the kern.ipc.nmbclusters to 2048 in the /boot/loader.conf, but when I checked netstat -m, it appears that there are less buffers available then there were when the problem happened.

netstat -m

enterprise# netstat -m
mbuf usage:
        GEN cache:      0/64 (in use/in pool)
        CPU #0 cache:   145/640 (in use/in pool)
        Total:          145/704 (in use/in pool)
        Mbuf cache high watermark: 512
        Maximum possible: 4096
        Allocated mbuf types:
          144 mbufs allocated to data
          1 mbufs allocated to packet headers
        17% of mbuf map consumed
mbuf cluster usage:
        GEN cache:      0/232 (in use/in pool)
        CPU #0 cache:   135/232 (in use/in pool)
        Total:          135/464 (in use/in pool)
        Cluster cache high watermark: 128
        Maximum possible: 2048                                                   <-- this number was much higher
        22% of cluster map consumed                                            <- this number was much lower.
1104 KBytes of wired memory reserved (27% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines

This particular machine has 512 MB of ram.

Any suggestions what an NFS intensive machine with 512 meg ram should have kern.ipc.nmbclusters set to?
Are there any otyher tunables I should be looking at.

-Grant